Search papers, labs, and topics across Lattice.
This paper introduces SpatioLM, a parameter-efficient model that enhances spatial reasoning in Vision-Language Models (VLMs) without relying on additional 3D inputs or external spatial encoders. By integrating a novel spatio-vision module and utilizing pseudo depth and camera information for supervision, SpatioLM effectively improves spatial perception and understanding while maintaining the general capabilities of VLMs. The model achieves a groundbreaking score of 71.6 on the VSI-Bench, marking the first instance of surpassing the 70-point threshold and demonstrating competitive performance in embodied manipulation tasks.
SpatioLM breaks new ground in spatial reasoning by achieving a record score on the VSI-Bench without the complexity of external spatial inputs.
Vision-Language Models (VLMs) perform well on commonsense reasoning tasks but struggle with visual spatial reasoning. Most existing solutions introduce extra 3D prior inputs or external spatial encoders, which increase complexity and degrade the underlying VLMs' general-purpose capabilities after spatial fine-tuning. To this end, we propose a parameter-efficient \textit{\textbf{Spatio}-vision \textbf{L}anguage \textbf{M}odels (SpatioLM)}, that enhances spatial intelligence without extra 3D prior inputs or third-party spatial encoders. Concretely, we design a plug-and-play and non-invasive spatio-vision module that elicits the spatial knowledge inherent in VLMs. Furthermore, we innovatively leverage pseudo depth and camera information as supervision to guide the model in learning physically coherent representations. Extensive experiments show that SpatioLM achieves significant improvements in diverse tasks, including spatial perception and understanding while effectively limiting the degradation of general capabilities. Notably, the model achieves an impressive score of 71.6 on the VSI-Bench (the first model to surpass 70). In addition, it attains competitive performance when transferred to embodied manipulation tasks. Code is available at \href{https://github.com/xiaomi-research/spatio-lm}{\faGithub~spatio-lm}.
