Search papers, labs, and topics across Lattice.
This paper introduces SEER, a novel inference-time evidence interface designed to enhance the accuracy of visual language models (VLMs) in spatial relation classification by making query-specific evidence explicit. By employing a method that hides candidate relations during pair localization and constructs a tailored view of subjects and objects, SEER significantly improves performance on relation-choice tasks, achieving gains of up to +11.79 across various models. The findings highlight the importance of query-specific evidence construction in mitigating common failures in VLMs, particularly in ambiguous scenarios.
Explicitly grounding evidence in spatial relation tasks can boost VLM performance by nearly 12 points, transforming how we approach visual reasoning.
Spatial relation questions require a model to identify the queried subject and object before comparing their layout. Yet a VLM can recognize both entities and still answer from the wrong instance or an ambiguous global view. We ask whether making query-specific evidence explicit can mitigate this failure and propose SEER (Self-grounded Evidence for Entity-Relation Reasoning), a training-free inference-time evidence interface for frozen VLMs. SEER hides candidate relations during pair localization, constructs a query-specific view with explicit subject/object roles, and retains the full image and sparse box geometry as complementary evidence. For relation-choice protocols with exact inverse support, an optional refinement swaps the entity roles and changes the forward decision only when exactly one visual state obeys the corresponding inverse relation. On an image-disjoint GQA-Train900 test frozen before model scoring, SEER pools to +3.94 [2.17,5.72] over Full; the gain remains positive under label-independent grounding-order counterbalancing and on the 535 rows whose entity names are unique. The unchanged protocol yields +4.35 to +11.79 on all 2,434 filtered EmbSpatial pair-relation questions across three models. Matched controls separate local refocus from role-explicit conditioning. These results establish query-specific evidence construction as the principal intervention, with reciprocal consistency as a smaller protocol-specific refinement.
