Search papers, labs, and topics across Lattice.
This study investigates the reasoning processes behind medical multimodal multiple-choice questions (MCQs) by distinguishing between correct answers supported by visual cues and those relying on non-visual information. The authors identify a phenomenon termed "reasoning inflation," where high benchmark scores do not accurately reflect the model's reasoning pathways. Through extensive analysis across six datasets, they reveal that while full-input accuracy is relatively high, the reliance on shortcuts significantly undermines the validity of medical image reasoning, as evidenced by the performance drop in a newly constructed subset, MedQA-MM.
Medical image reasoning claims are often inflated by shortcuts, with accuracy plummeting to 5.21% when these cues are mitigated.
A benchmark score credits final answers, but not the route by which an item can be answered. In medical multimodal multiple-choice questions (MCQs), this distinction matters because a correct answer can be supported by the intended image finding or by benchmark-preserved cues in the wording of answers, non-visual clinical text, visible image text, artificial annotations, or device/context artifacts. We call the resulting score-level overinterpretation reasoning inflation. Here, a route is an observable input path that can support answer selection, not a claim about the model's hidden cognition. Across six medical multimodal MCQ datasets, we separate candidate cues from behavioral evidence through prompt- and image-side audits, modality ablations, and matched repairs that preserve the medical target and answer key. In a 13-configuration open-model panel, full-input accuracy is 62.63%, while text-only and options-only settings achieve 53.96% and 29.71%, respectively. Removing length-gap, absolute/conspicuous, and spatial/prepositional cues lowers accuracy by 6.58, 3.50, and 4.77 percentage points. We also construct MedQA-MM, a 1,000-item shortcut-mitigated subset, where text-only and options-only accuracy fall to 5.21% and 12.33%. This does not imply that models never use images; it shows that medical image-reasoning claims require route-level evidence.
