Search papers, labs, and topics across Lattice.
This paper introduces SAVER, a rule-based method that enhances the accuracy of vision-language models (VLMs) in visual change reasoning by selectively auditing verbal evidence in their outputs. The authors demonstrate that correct outputs are often supported by explicit verbal evidence, while incorrect ones lack this crucial information, leading to a structured reprompting approach when evidence is missing or inconsistent. Evaluated across three benchmarks, SAVER achieves significant accuracy improvements, with a notable gain of up to 25.8% on the CLEVR-Change dataset, highlighting the importance of articulating visual observations for effective reasoning.
Missing verbal evidence in VLM outputs can lead to significant reasoning errors, but SAVER recovers accuracy by enforcing explicit articulation of visual changes.
Vision-language models (VLMs) frequently fail at visual change reasoning, even when their vision encoders contain sufficient information. We observe that correct VLM outputs tend to contain explicit verbal evidence (object names, colors, spatial locations) that supports the claimed change, while incorrect outputs often lack such evidence. We propose SAVER (Selective Auditing of Verbal Evidence for Error Recovery), a lightweight, rule-based method that parses VLM responses for this evidence and triggers structured reprompting only when evidence is missing or inconsistent. Across three change detection benchmarks and four VLMs, SAVER significantly improves accuracy on tasks where errors stem from the model failing to articulate what it saw (expression failures), with gains up to +25.8% on CLEVR-Change. The evidence patterns can also be generated by an LLM in a single call, matching the hand-tuned gate on CLEVR-Change. Ablation experiments confirm that the evidence gate, not reprompting alone, drives the improvement.
