Search papers, labs, and topics across Lattice.
This paper introduces VLM-IE3D, a novel framework that enhances vision-language models (VLMs) by integrating both implicit and explicit 3D geometries derived from RGB videos to improve spatial reasoning and understanding. The framework employs Implicit Geometry Tokens (IGTs) for high-level geometric priors and Explicit Geometry Tokens (EGTs) for detailed geometric structures, effectively merging these with 2D visual cues through a 3D-aware adapter. Experimental results demonstrate that VLM-IE3D significantly outperforms existing models across a range of 3D tasks, including detection, grounding, and dense captioning.
VLM-IE3D achieves state-of-the-art performance in 3D tasks by seamlessly integrating implicit and explicit geometric representations from RGB inputs.
Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs often struggle when handling various 3D tasks that require fine-grained spatial understanding and reasoning. To bridge this gap, we present VLM-IE3D, a unified framework that enhances the 3D spatial awareness of VLMs by equipping them with both implicit and explicit 3D geometries learned from RGB videos. Our VLM-IE3D introduces Implicit Geometry Tokens (IGTs) that capture high-level geometric priors from input videos, as well as complementary Explicit Geometry Tokens (EGTs) that encode detailed geometric structures from reconstructed 3D attributes. On top of that, VLM-IE3D comes with a 3D-aware adapter that effectively fuses the two types of geometric representations with 2D visual cues. This RGB-only design injects strong 3D inductive biases for fine-grained spatial understanding and reasoning without requiring any additional 3D inputs. Extensive experiments show that VLM-IE3D achieves superior performance consistently across various 3D tasks including 3D video detection, 3D visual grounding, 3D dense captioning, and spatial reasoning. Code and models are available at https://github.com/Vegetebird/VLM-IE3D.
DAMO