Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors mechanistically dissect relative-position spatial reasoning across three vision-language models and their LLM backbones, isolating how models track object locations versus bind entities to query roles. Causal activation patching reveals a consistent, staged circuit spanning early-layer visual source representations, intermediate-layer query-object mentions, and late-layer relational decision states. Exploiting an identified linear role-binding direction allows representation steering derived from synthetic scenes to transfer directly to natural-image benchmarks, significantly boosting pairwise consistency and accuracy without fine-tuning.
Spatial inconsistency in VLMs stems from a dissociable role-binding circuit that can be linearly steered to fix swapped-object reasoning failures on natural images without fine-tuning.
High instance-level accuracy can mask inconsistencies in spatial reasoning when objects exchange positions or their roles are reversed in the query. The internal representations supporting relative-position reasoning remain poorly understood. We investigate two complementary components of this process: tracking object locations in the input and representing their query roles. Across three VLMs with visual or textual inputs and their language-model backbones, activation patching reveals a staged progression from early-layer source representations through intermediate-layer query-object representations to late-layer answer states. Targeted interventions further establish causal links along this progression: manipulating source-side representations shifts location information at query-object mentions and ultimately alters relation predictions. Beyond object-location information, we also identify a stable query-side direction associated with the roles of the two objects in the comparison. Steering along directions estimated on synthetic scenes generalizes to natural-image benchmarks, improving accuracy and both forms of paired consistency in most settings without retraining. Our findings reveal complementary components of relational reasoning across visual and textual settings and show how targeted interventions can improve the consistency of models' behavior.