Search papers, labs, and topics across Lattice.
The paper introduces GeoAlign, a method to improve spatial reasoning in MLLMs by dynamically aggregating multi-layer geometric features from 3D foundation models. It addresses the task misalignment bias of using static, single-layer geometric features by constructing a hierarchical feature bank and using MLLM visual tokens to perform content-aware, layer-wise sparse routing. Experiments show GeoAlign achieves state-of-the-art performance on spatial reasoning benchmarks, even surpassing larger MLLMs with a 4B model.
Static geometric features are holding back MLLMs' spatial reasoning abilities; GeoAlign's dynamic, content-aware routing of multi-layer features unlocks SOTA performance with a compact model.
Multimodal large language models (MLLMs) have exhibited remarkable performance in various visual tasks, yet still struggle with spatial reasoning. Recent efforts mitigate this by injecting geometric features from 3D foundation models, but rely on static single-layer extractions. We identify that such an approach induces a task misalignment bias: the geometric features naturally evolve towards 3D pretraining objectives, which may contradict the heterogeneous spatial demands of MLLMs, rendering any single layer fundamentally insufficient. To resolve this, we propose GeoAlign, a novel framework that dynamically aggregates multi-layer geometric features to realign with the actual demands. GeoAlign constructs a hierarchical geometric feature bank and leverages the MLLM's original visual tokens as content-aware queries to perform layer-wise sparse routing, adaptively fetching the suitable geometric features for each patch. Extensive experiments on VSI-Bench, ScanQA, and SQA3D demonstrate that our compact 4B model effectively achieves state-of-the-art performance, even outperforming larger existing MLLMs.
