Search papers, labs, and topics across Lattice.
This study investigates the relationship between object localization and spatial reasoning in vision-language models (VLMs) by analyzing LLaVA-1.5 and Qwen2.5-VL using mechanistic interpretability tools. The findings reveal that spatial reasoning does not require precise object localization but instead utilizes coarse target-reference anchors established by object-aligned tokens, with positional information becoming decodable prior to relation decisions. The research highlights that while grounding and reasoning share early processing stages, they ultimately depend on distinct pathways, challenging the assumption that knowing object locations directly translates to understanding their spatial relationships.
Spatial reasoning in VLMs operates on coarse object localization rather than precise boundaries, revealing a surprising disconnect between knowing where objects are and how they relate.
Vision-language models (VLMs) can answer spatial questions, yet the mechanisms connecting object grounding to spatial reasoning remain poorly understood. It is underexplored whether spatial reasoning internally requires precise objects localization, or can bypass explicit localization through global layout cues. In this work, we investigate two representative model families, LLaVA-1.5 and Qwen2.5-VL, using a suite of mechanistic interpretability tools, including token ablation, layer-wise probing, attention knockout, and causal mediation analysis. We find that spatial relation prediction follows a staged grounding-to-reasoning process in which object-aligned tokens establish coarse target-reference anchors, while precise bounding-box boundaries are not required. Positional information becomes decodable before relation decisions emerge, and a small set of attention heads mediates the causal effects of both localization and spatial reasoning. The two tasks share early grounding-related processing but ultimately rely on partially distinct specialized pathways. Through rigorous experiments, we provide a token-, layer-, and head-level account of how VLMs transform object grounding into spatial relations, showing that knowing where objects are is not equivalent to knowing how they relate.
