Search papers, labs, and topics across Lattice.
This study evaluates the reliance on visual evidence in traffic accident Video Question Answering (VideoQA) by auditing four public benchmarks and revealing that many Vision-Language Models (VLMs) achieve high accuracy without utilizing video input. The authors introduce metrics such as Blind Gap and Visual Gain to quantify visual dependence, and develop a Shortcut Score for filtering questions that rely on textual shortcuts rather than visual context. Their findings highlight significant discrepancies in grounding quality across benchmarks, emphasizing the need for visually grounded evaluation in safety-critical applications.
High accuracy in VideoQA can be misleading, as many models perform better without visual input, revealing a dangerous reliance on textual shortcuts.
High benchmark accuracy does not guarantee genuine use of visual evidence. We study this problem in traffic accident Video Question Answering (VideoQA), where correct answers should depend on scene-specific visual evidence but may instead be inferred from textual shortcuts. Through an audit of four public benchmarks, we find that several recent open-weight Vision-Language Models (VLMs) perform competitively, and sometimes better, without video input. On the MM-AU benchmark, removing video consistently improves accuracy, and adding more frames further degrades performance. To quantify visual dependence, we introduce two dataset-level diagnostics: Blind Gap, measuring above-chance text-only performance, and Visual Gain, measuring the marginal benefit of adding video. We further propose an instance-level Shortcut Score that combines text-only confidence with visual necessity signals, enabling continuous, training-free filtering of shortcut-prone questions. The resulting subsets reduce shortcut bias and improve visual grounding. Our findings reveal large differences in grounding quality across benchmarks and show that visually grounded evaluation, not just high accuracy, is essential in safety-critical VideoQA.
