Search papers, labs, and topics across Lattice.
This paper introduces SpatialSV, a novel framework that enhances multimodal large language models (MLLMs) by internalizing 3D spatial awareness through task-oriented visual supervision. Unlike traditional methods that rely on external tools or latent feature distillation, SpatialSV actively transforms 2D visual features into interpretable 3D representations, such as depth maps and point clouds, providing a clear insight into the model's spatial understanding. Experimental results show that SpatialSV significantly improves the spatial intelligence of MLLMs while maintaining strong generalization capabilities in semi-supervised contexts.
Transforming 2D visual features into interpretable 3D representations unlocks a new level of spatial intelligence in MLLMs, offering unprecedented insights into their internal workings.
Unlocking the spatial intelligence of multimodal large language model (MLLMs) is crucial for understanding and interacting with the 3D world. Prevailing approaches typically inject spatial priors via external tools, which impose significant inference overhead, or rely on latent feature distillation, which remains uninterpretable and lacks fine-grained geometric constraints. To address these issues, we propose SpatialSV, a framework designed to internalize robust 3D spatial awareness within MLLMs while simultaneously offering inherent interpretability. Deviating from passive feature imitation, SpatialSV employs task-oriented visual supervision, compelling the model to actively lift its 2D visual features into explicit 3D representations, including depth maps, camera poses, and point clouds. Crucially, this 2D-to-3D lifting process provides a transparent window into the model's representations: the resulting 3D reconstructions serve as an intuitive proxy for visualizing and diagnosing the quality of the model's intrinsic spatial knowledge. Extensive experiments across multiple models and benchmarks demonstrate the effectiveness of SpatialSV in enhancing and interpreting MLLMs' spatial intelligence. Furthermore, the framework exhibits strong generalization in semi-supervised settings, validating its potential to leverage unlabeled visual data for scalable, interpretable spatial representation learning.
