Search papers, labs, and topics across Lattice.
This study investigates how vision-language models (VLMs) utilize textual shortcuts from prior chains of thought (CoT) instead of recomputing visual evidence when reasoning about images. Through a matched counterfactual analysis across 16 VLMs, the authors reveal that evidence-bearing content significantly influences model behavior, with its removal leading to a notable shift in answer preference. To mitigate this reliance on stale information, they introduce the Fresh-State Attention Firewall (FSAF), which effectively increases visual update rates and reduces the frequency of prior-answer reuse in VLMs.
Textual shortcuts in VLMs can significantly undermine reasoning accuracy, but a simple intervention can restore fresh visual computation.
Vision-language models (VLMs) are expected to revise their reasoning when visual evidence changes. Failures to do so are often attributed to insufficient visual attention or contextual inertia, leaving unclear what models reuse instead of recomputing from the current image. We show that evidence-bearing reasoning in a prior chain of thought (CoT) can form a textual shortcut that competes behaviorally with visual recomputation. Across 16 VLMs, a matched counterfactual analysis identifies evidence-bearing content as the most robust carrier of prior-CoT influence. Removing this evidence-bearing content shifts answer preference more than removing length-matched non-evidence context or the final-answer span, with prior control weakening progressively as more stale evidence is removed. Reordering this evidence also weakens prior control, showing that its organization modulates shortcut strength. Beyond the immediate answer, the shortcut can retain residual influence after answer correction: weakening current-image support shifts preference back toward the prior answer, while repeated prior answers and reused premises arise mainly when the shortcut remains active. To limit this influence, we introduce Fresh-State Attention Firewall (FSAF), a training-free intervention that isolates fresh computation from the prior CoT. Across five VLMs, FSAF raises visual update rate from 35.28% to 53.61% and reduces prior-answer rate from 39.22% to 3.67%. Reliable VLM self-reflection therefore requires more than looking again: fresh visual recomputation must be protected from stale textual reuse.
