Search papers, labs, and topics across Lattice.
This paper introduces Geometrically Referenced 3D scene representations (GR3D), a novel approach to enhance MLLMs' 3D spatial reasoning capabilities. GR3D annotates objects in input images with unique IDs and encodes their 3D geometric attributes as textual references, enabling MLLMs to leverage language-based mathematical reasoning alongside 2D visual features. Experiments in a zero-shot setting demonstrate that GR3D boosts GPT-5's performance on VSI-Bench by 8% overall, with even greater improvements on tasks requiring spatial layout understanding.
MLLMs can gain a surprising boost in 3D spatial reasoning simply by encoding 3D geometric attributes of objects as textual references indexed by unique IDs.
While Multimodal Large Language Models (MLLMs) have achieved remarkable success in 2D visual understanding, their ability to reason about 3D space remains limited. To address this gap, we introduce geometrically referenced 3D scene representations (GR3D). Given a set of input images, GR3D annotates objects in the images with unique IDs and encodes their 3D geometric attributes as textual references indexed by these IDs. This representation enables MLLMs to interpret 3D cues using their advanced language-based skills in mathematical reasoning, while concurrently analyzing 2D visual features in a tightly coupled way. We present a simple yet effective approach based on GR3D, which requires no additional training and is readily applicable to different MLLMs. Implemented in a zero-shot setting, our approach boosts GPT-5's performance on VSI-Bench by 8% overall and more than 11% on tasks that rely heavily on spatial layout understanding. Qualitative studies further demonstrate that GR3D empowers MLLMs to perform complex spatial reasoning with highly sparse input views.
