Search papers, labs, and topics across Lattice.
This paper introduces ViPS, a multi-model prior framework that integrates diverse visual priors from multiple foundation models to enhance spatial understanding in Multimodal Large Language Models (MLLMs). By employing an Efficient Prior Proxy for generating foundational priors and a Dynamic Prior Fusion mechanism for context-aware integration, ViPS effectively harmonizes these diverse inputs to improve performance on spatial reasoning tasks. Experimental results show that ViPS achieves new state-of-the-art results across various benchmarks, highlighting the benefits of leveraging complementary spatial knowledge from different models.
Integrating diverse visual priors can lead to significant improvements in spatial reasoning tasks, with ViPS setting new benchmarks in MLLM performance.
Multimodal Large Language Models (MLLMs) have demonstrated substantial promise in spatial understanding. Existing works typically incorporate prior knowledge extracted from a pre-trained foundation model to further enhance the spatial awareness of MLLMs. In this paper, we first reveal that when integrating diverse foundation models into MLLMs, different models provide complementary spatial priors that benefit different tasks. Motivated by this, we propose $\textbf{ViPS}$, a novel multi-model prior framework designed to fully unleash the potential of incorporating multiple $\textbf{Vi}$sual $\textbf{P}$riors from diverse models into MLLMs for $\textbf{S}$patial understanding. Specifically, ViPS introduces an Efficient Prior Proxy to generate multiple foundational priors with minimal inference overhead, and a Dynamic Prior Fusion mechanism to achieve harmonious and context-aware prior fusion and injection from the prior proxies. Extensive experiments demonstrate that ViPS successfully harmonizes diverse visual priors, establishing new state-of-the-art performance across multiple complex spatial reasoning and 3D spatial understanding benchmarks. Project page: https://visual-ai.github.io/vips
