Search papers, labs, and topics across Lattice.
This paper introduces Occ-VLM, a framework that enhances 3D scene understanding by utilizing only posed RGB images and a single 2D vision encoder, thereby integrating 3D geometric perception with 2D semantics. By reconstructing 3D scene occupancy as an auxiliary geometric prior, Occ-VLM spatially associates foreground 2D tokens with 3D space, allowing for effective decoding by a Large Language Model (LLM). The results show that Occ-VLM achieves state-of-the-art performance in multi-view occupancy prediction and matches the performance of traditional 3D-input VLMs in 3D Visual Question Answering and dense captioning tasks.
Achieving state-of-the-art occupancy prediction while using only 2D images, Occ-VLM bridges the gap between 2D semantics and 3D understanding without the need for complex 3D inputs.
Recently, vision-language models (VLMs) have made significant progress in 3D scene understanding, driving advances in applications such as embodied intelligence and robotic vision. However, existing approaches typically either rely directly on explicit 3D inputs (e.g., point clouds or RGB-D sequences), or introduce an additional 3D geometry encoder to derive 3D-aware visual tokens from 2D images. Such designs structurally decouple 3D geometric perception from the rich 2D semantics learned via vision-language pre-training, hindering the development of a unified 3D vision-language representation. In this work, we propose Occ-VLM, a novel framework for 3D scene understanding that operates purely on posed RGB images and employs a single 2D vision encoder. Specifically, Occ-VLM reconstructs 3D scene occupancy as an auxiliary geometric prior, which is utilized to spatially associate foreground 2D tokens with 3D space. These tokens are then decoded by a Large Language Model (LLM) for unified scene understanding. Extensive experiments demonstrate that Occ-VLM achieves both accurate geometric perception and robust vision-language reasoning: it attains state-of-the-art performance on multi-view occupancy prediction, while performing on par with 3D-input VLMs on 3D Visual Question Answering (VQA) and 3D dense captioning benchmarks.
