Search papers, labs, and topics across Lattice.
The paper introduces GR3D, a spatial vision-language model with explicit 2D, implicit 2D, and monocular 3D grounding capabilities. GR3D uses an implicit grounding mechanism to identify entity mentions and insert corresponding region tokens into the text stream, enabling reference to visual evidence during spatial chain-of-thought reasoning. The model also employs a region-prompted monocular 3D grounding design to predict 3D bounding boxes, achieving improved performance on spatial benchmarks and demonstrating grounding as a beneficial inductive bias.
Grounding boosts spatial reasoning in VLMs: explicitly linking language to 2D and 3D scene elements lets models decompose complex spatial problems and improve performance even on non-grounded tasks.
We present GR3D, a spatial vision language model equipped with three complementary grounding capabilities--explicit 2D grounding, implicit 2D grounding, and monocular 3D grounding--within a single framework. GR3D introduces an implicit grounding mechanism that identifies entity mentions during generation and inserts the corresponding region tokens into the text stream, allowing the model to reference visual evidence on the fly when producing spatial chain-of-thought responses. In parallel, a region-prompted monocular 3D grounding design predicts 3D bounding boxes in the camera view from grounded region queries, supported by intrinsic-aware normalization and dense geometric supervision. Together, these grounding capabilities enable GR3D to decompose complex spatial understanding problems into grounded 2D perception followed by 3D inference. GR3D achieves consistent improvements across grounded and non-grounded spatial benchmarks, demonstrating grounding as an effective inductive bias for strengthening spatial understanding in VLMs. These grounding capabilities collectively enhance general spatial understanding beyond the grounding task itself.
NVIDIA