Search papers, labs, and topics across Lattice.
This paper introduces Counterfactual Anatomy-guided Spatial-Temporal decoding (CAST), a novel framework for mitigating hallucinations in medical vision-language models (Med-VLMs) without the need for manual annotations. By leveraging broad medical segmentation to identify relevant anatomical regions and employing counterfactual interventions to enhance spatial attention, CAST effectively improves the accuracy of generated responses. Experimental results on the SLAKE and MIMIC-CXR datasets show that CAST significantly outperforms existing methods, demonstrating its potential as a robust solution for enhancing the reliability of Med-VLMs in clinical settings.
Compact, automatically selected anatomical regions can drastically reduce hallucinations in medical VLMs without requiring expert annotations.
Medical vision-language models (Med-VLMs) have demonstrated strong performance on medical visual question answering, yet they remain prone to hallucination, generating clinically unsupported statements that are insufficiently grounded in image evidence. Mitigation methods applied during decoding offer a practical solution, but they typically lack anatomical awareness or rely heavily on ground truth annotations, which limits their applicability. We propose Counterfactual Anatomy-guided Spatial-Temporal decoding (CAST), a framework that operates entirely during inference and requires no manual annotations for anatomically grounded hallucination mitigation. CAST automatically discovers anatomical regions relevant to the given query through broad medical segmentation. It then selects a compact, causally informative area using counterfactual intervention based on the drop in answer likelihood under occlusion. Guided by this chosen region, CAST performs a unified contrastive decoding process, combining classifier-free guidance to correct spatial attention with stepwise temporal contrast to regulate generation dynamics. Experiments on the SLAKE and MIMIC-CXR datasets across three Med-VLMs demonstrate that CAST consistently outperforms strong baselines and surpasses decoding strategies reliant on ground truth. Our results indicate that compact, automatically selected regions provide highly effective contrastive guidance without expert annotations, offering a practical and generalizable solution for improving spatial grounding and reducing hallucinations. Code is available at https://github.com/csyifan/CAST.
