Search papers, labs, and topics across Lattice.
The paper introduces Dynamic Decision Learning (DDL), a framework that iteratively refines abnormality grounding in rare disease imaging using frozen vision-language models. DDL optimizes instructions and consolidates predictions under visual perturbations to improve localization quality and generate a consensus-based reliability score. Experiments on brain imaging datasets, including a rare-disease dataset, demonstrate that DDL significantly improves mAP@75, outperforming adaptation baselines and supervised fine-tuning, particularly in rare-disease cases.
Frozen vision-language models can dramatically improve abnormality grounding in rare disease imaging by iteratively refining decisions through optimized instructions and visual perturbations.
Clinical abnormality grounding for rare diseases is often hindered by data scarcity, making supervised fine-tuning impractical and single-pass inference highly unstable. We propose Dynamic Decision Learning (DDL), a framework that enables frozen large vision-language models (LVLMs) to refine their decisions across both language and visual spaces by optimizing instructions and consolidating predictions under visual perturbations. This process improves localization quality and produces a consensus-based reliability score that quantifies model confidence. Results on brain imaging benchmarks, including a rare-disease dataset with 281 pathology types across models ranging from 3B to 72B parameters, show that DDL improves mAP@75 by up to 105% on rare-disease cases and outperforms adaptation baselines and supervised fine-tuning. Furthermore, DDL demonstrates stronger calibration between reliability scores and localization accuracy under severe distribution shifts and increasing task difficulty. Code is available at: https://lijunrio.github.io/DDL/
Tsinghua AI