Search papers, labs, and topics across Lattice.
The paper introduces SpaceMind, a vision-language model designed for spatial reasoning from RGB inputs, addressing limitations in existing VLMs that struggle with 3D spatial tasks. SpaceMind employs a dual-encoder architecture with VGGT for spatial understanding and InternViT for 2D visual encoding, using a novel Camera-Guided Modality Fusion module to explicitly leverage camera parameters. Experiments demonstrate that SpaceMind achieves state-of-the-art results on VSI-Bench, SQA3D, and SPBench, indicating the effectiveness of camera-guided modality fusion for spatial reasoning.
By treating camera parameters as an active guide rather than mere metadata, SpaceMind significantly boosts VLMs' spatial reasoning abilities from RGB images alone.
Large vision-language models (VLMs) show strong multimodal understanding but still struggle with 3D spatial reasoning, such as distance estimation, size comparison, and cross-view consistency. Existing 3D-aware methods either depend on auxiliary 3D information or enhance RGB-only VLMs with geometry encoders through shallow feature fusion. We propose SpaceMind, a multimodal large language model explicitly designed for spatial reasoning solely from RGB inputs. The model adopts a dual-encoder architecture, integrating VGGT as a spatial understanding encoder and InternViT as a 2D visual encoder. The key idea is to treat the camera representation as an active guiding modality rather than passive metadata. Specifically, SpaceMind introduces a lightweight Camera-Guided Modality Fusion module before the language model to replace shallow fusion. It applies camera-conditioned biasing to spatial tokens, assigns query-independent weights reflecting their geometric importance, and uses the camera embedding to gate the fused representation. Empirically, SpaceMind establishes new state-of-the-art results on VSI-Bench, SQA3D and SPBench, surpassing both open and proprietary systems on VSI-Bench and SPBench by large margins and achieving state-of-the-art performance on SQA3D. These results demonstrate that camera-guided modality fusion is an effective and practical inductive bias for equipping VLMs with genuinely spatially grounded intelligence. We will release code and model checkpoints to support future research.
