Search papers, labs, and topics across Lattice.
This paper investigates how question framing impacts visual attention and reasoning in VLMs, revealing that constrained framings (e.g., multiple choice) significantly reduce attention to visual inputs compared to open-ended questions. By quantifying attention allocation, the authors show that framing biases attention away from task-relevant regions and towards uninformative tokens, directly causing performance degradation and inconsistency across different question types. They then introduce a prompt-tuning method that encourages visually grounded attention, leading to improved performance across framings.
VLMs selectively ignore visual information based on question framing, even when the visual reasoning task remains identical, highlighting a critical vulnerability in their grounding capabilities.
Vision-Language Models (VLMs) have been shown to be blind, often underutilizing their visual inputs even on tasks that require visual reasoning. In this work, we demonstrate that VLMs are selectively blind. They modulate the amount of attention applied to visual inputs based on linguistic framing even when alternative framings demand identical visual reasoning. Using visual attention as a probe, we quantify how framing alters both the amount and distribution of attention over the image. Constrained framings, such as multiple choice and yes/no, induce substantially lower attention to image context compared to open-ended, reduce focus on task-relevant regions, and shift attention towards uninformative tokens. We further demonstrate that this attention misallocation is the principal cause of degraded accuracy and cross-framing inconsistency. Building on this mechanistic insight, we introduce a lightweight prompt-tuning method using learnable tokens that encourages the robust, visually grounded attention patterns observed in open-ended settings, improving visual grounding and improving performance across framings.
