Search papers, labs, and topics across Lattice.
The paper introduces GAP-MLLM, a geometry-aligned pre-training paradigm for multimodal large language models (MLLMs) to improve 3D spatial perception from RGB inputs. It addresses the misalignment between text-dominated fine-tuning and geometric representation by pre-training MLLMs to predict sparse pointmaps and semantic labels, enforcing geometric awareness. The method also includes a multi-level progressive fusion module with token-level gating for adaptive integration of geometric priors, leading to performance gains in 3D visual grounding, dense captioning, and video object detection.
Text-heavy fine-tuning is blinding your MLLM to crucial 3D spatial information, but GAP-MLLM's geometry-aligned pre-training can restore its sight.
Multimodal Large Language Models (MLLMs) demonstrate exceptional semantic reasoning but struggle with 3D spatial perception when restricted to pure RGB inputs. Despite leveraging implicit geometric priors from 3D reconstruction models, image-based methods still exhibit a notable performance gap compared to methods using explicit 3D data. We argue that this gap does not arise from insufficient geometric priors, but from a misalignment in the training paradigm: text-dominated fine-tuning fails to activate geometric representations within MLLMs. Existing approaches typically resort to naive feature concatenation and optimize directly for downstream tasks without geometry-specific supervision, leading to suboptimal structural utilization. To address this limitation, we propose GAP-MLLM, a Geometry-Aligned Pre-training paradigm that explicitly activates structural perception before downstream adaptation. Specifically, we introduce a visual-prompted joint task that compels the MLLMs to predict sparse pointmaps alongside semantic labels, thereby enforcing geometric awareness. Furthermore, we design a multi-level progressive fusion module with a token-level gating mechanism, enabling adaptive integration of geometric priors without suppressing semantic reasoning. Extensive experiments demonstrate that GAP-MLLM significantly enhances geometric feature fusion and consistently enhances performance across 3D visual grounding, 3D dense captioning, and 3D video object detection tasks.
