Search papers, labs, and topics across Lattice.
This paper introduces SHOVIR, a benchmark designed to evaluate vision shortcut learning in radiology report generation by extending existing chest X-ray datasets with per-box CheXpert labels and conducting image-level and disease-level occlusion experiments. The study reveals that while some vision-language models (VLMs) achieve high report quality, they often rely on spurious correlations, leading to significant variations in spatial grounding across different architectures and datasets. By isolating failure modes such as direct and contextual shortcuts, the findings highlight a critical gap in current evaluation protocols that could misrepresent a model's true diagnostic capabilities.
High report quality in VLMs can mask a dangerous reliance on visual shortcuts, revealing a critical flaw in how we evaluate radiology report generation.
Current evaluation protocols for Vision-Language Models (VLMs) in Radiology Report Generation (RRG) rely on report-level metrics that measure lexical overlap or aggregate clinical correctness. However, such metrics do not test whether individual diagnostic statements stem from the actual pathological evidence visible in the image. This allows models to achieve competitive scores by exploiting learned priors or spurious correlations, a failure mode we refer to as vision shortcut. We introduce SHOVIR, a benchmark for evaluating vision shortcut behavior in RRG. SHOVIR extends two spatially annotated chest X-ray datasets, MIMIC-CXR and PadChest-GR, with per-box CheXpert labels, and defines image-level and disease-level occlusion experiments that contrast baseline performance on clean images against localized, region-specific perturbations. Comparing predictions across these conditions isolates two failure modes at the disease-class level: direct shortcuts, where a finding persists after its visual evidence is removed, and contextual shortcuts, where detection degrades once co-occurring pathologies are occluded despite the target region remaining intact. Benchmarking eight state-of-the-art VLMs, we find that shortcut behavior varies substantially across architectures and datasets. Models achieving the highest baseline report quality do not necessarily rank highest in spatial grounding, revealing that clinically fluent generation can coexist with shallow reliance on visual evidence. These findings expose a blind spot in current RRG evaluation and motivate region-aware assessment protocols.
