Search papers, labs, and topics across Lattice.
This paper introduces Composer, a Multimodal Large Language Model (MLLM) that utilizes learned proxy-tokens for improved visual grounding in Visual Question Answering (VQA). By addressing the semantic-spatial gap present in traditional grounding methods, Composer enables more faithful interpretability by allowing the model to manipulate visual regions as semantically meaningful entities. Experimental results show that Composer matches the answer accuracy of existing coordinate-based models while enhancing visual grounding accuracy by 9.0 points, highlighting the effectiveness of discrete proxy-tokens in capturing spatial semantics.
Discrete proxy-tokens in Composer enhance visual grounding accuracy by 9.0 points, bridging the semantic-spatial gap that plagues traditional methods.
Multimodal Large Language Models (MLLMs) have achieved remarkable success in Visual Question Answering (VQA), yet their "black-box" nature hinders deployment in critical domains. Grounded Visual Reasoning (GVR) approaches attempt to improve interpretability by explicitly couple textual rationales with visual grounding information, which are typically textual coordinates. This mechanism lacks a learnable semantic link to the visual features, often resulting in a semantic-spatial gap where the model hallucinates coordinates that do not correspond to image evidences. In this work, we introduce Composer, a MLLM that leverages a novel visual grounding mechanism based on learned proxy-tokens to promote faithful interpretability. These discrete symbolic pointers explicitly index the image latent space, allowing the model to manipulate visual regions as addressable, semantically manipulable sets. To rigorously validate our novel grounding mechanism, we constructed ComposerGCoT, a dataset synthesized to enable holistic assessment of reasoning consistency and grounding accuracy. Experimental results indicate that Composer achieves performance parity with its coordinate-based counterpart in final answer accuracy, while improving visual grounding accuracy by +9.0 points. By demonstrating that discrete proxy-tokens capture spatial semantics more effectively than typical textual coordinates, we establish that visual grounding mechanisms with learnable semantic links represent a promising path toward trustworthy and reliable MLLMs.
