Search papers, labs, and topics across Lattice.
The paper introduces SpatialGeo, a novel vision encoder that enhances spatial reasoning in MLLMs by fusing geometric and semantic features. They identify that the spatial reasoning limitations of existing MLLMs arise from the lossy embedding of vision encoders like CLIP, which are restricted to instance-level semantic features. SpatialGeo addresses this by incorporating geometry features from vision-only self-supervised learning via a hierarchical adapter, leading to improved spatial awareness and reduced memory cost.
MLLMs can get a whopping 8% boost in spatial reasoning accuracy, with 50% less memory, simply by fusing geometric and semantic features in the vision encoder.
Multimodal large language models (MLLMs) have achieved significant progress in image and language tasks due to the strong reasoning capability of large language models (LLMs). Nevertheless, most MLLMs suffer from limited spatial reasoning ability to interpret and infer spatial arrangements in three-dimensional space. In this work, we propose a novel vision encoder based on hierarchical fusion of geometry and semantics features, generating spatial-aware visual embedding and boosting the spatial grounding capability of MLLMs. Specifically, we first unveil that the spatial ambiguity shortcoming stems from the lossy embedding of the vision encoder utilized in most existing MLLMs (e.g., CLIP), restricted to instance-level semantic features. This motivates us to complement CLIP with the geometry features from vision-only self-supervised learning via a hierarchical adapter, enhancing the spatial awareness in the proposed SpatialGeo. The network is efficiently trained using pretrained LLaVA model and optimized with random feature dropping to avoid trivial solutions relying solely on the CLIP encoder. Experimental results show that SpatialGeo improves the accuracy in spatial reasoning tasks, enhancing state-of-the-art models by at least 8.0% in SpatialRGPT-Bench with ∼50% less memory cost during inference. The source code is available via https://ricky-plus.github.io/SpatialGeoPages/.
