Search papers, labs, and topics across Lattice.
This study identifies Visual Retrieval Heads (VRHs) in vision-language models (VLMs) that are crucial for linking text prompts to specific image regions, revealing that only a small fraction of attention heads (1.7-2.6%) are responsible for this task. By employing a novel scoring method that evaluates attention from output prediction tokens against ground-truth regions, the researchers demonstrate that masking these VRHs can drastically impair grounding accuracy by up to 80 percentage points. The findings not only replicate established principles from language models but also highlight the unique properties of VRHs, such as their generalization across various visual tasks and their functional specificity.
Masking just 20 Visual Retrieval Heads in VLMs can lead to an 80-point drop in grounding accuracy, revealing their critical role in visual information extraction.
Vision-language models (VLMs) can locate an image region referred to by a text prompt and route the corresponding visual evidence to the output, yet the internal mechanism behind this behavior is not understood. Inspired by retrieval heads in large language models, we ask whether VLMs contain an analogous mechanism for visual retrieval. We answer affirmatively by introducing Visual Retrieval Heads (VRHs), a small subset of attention heads (about 1.7-2.6%) that are causally responsible for grounding text descriptions to image regions. To find them, we recast existing head-scoring methods under a unified design space over query tokens, key aggregation, and cross-sample aggregation. We then show that scoring attention from output prediction tokens with a sum over the ground-truth referent region most reliably identifies causal heads. Across eleven VLMs and five referring-expression benchmarks, masking only the top 20 VRHs reduces grounding accuracy by up to 80 percentage points, while masking the same number of random heads has little effect. Beyond replicating the causal-sparse-universal triad established for text retrieval heads, VRHs exhibit several properties not previously reported: they generalize across visual reference tasks, remaining causal on attribute, spatial, counting, and visual-math benchmarks despite being discovered through bounding-box prediction; they are functionally specific, preserving output format while corrupting localization; and they are architecturally shared, transferring causally across VLMs that share an LLM backbone but differ in vision encoder, projector, and instruction tuning.
