Search papers, labs, and topics across Lattice.
This paper introduces VGOcc, a novel approach for vision-only 3D occupancy prediction that leverages both visual and geometric cues from foundation models to enhance Gaussian occupancy modeling. By integrating richer geometric information with sparse Gaussian primitives, VGOcc improves the initialization and refinement of occupancy representations, resulting in a method that effectively captures semantic occupancy from calibrated surround-view images. Experimental results on the nuScenes dataset show that VGOcc achieves state-of-the-art performance, significantly advancing the field of 3D scene representation.
Visual-Geometric Gaussians redefine occupancy prediction by integrating geometric insights with traditional image features, leading to unprecedented accuracy in 3D scene understanding.
Vision-only occupancy prediction requires recovering a semantic 3D occupancy field from calibrated surround-view images, where each view provides observations with ambiguous depth along camera rays. Existing methods have progressed from dense structured representations to sparse Gaussian primitives, improving the efficiency of 3D scene representation. However, Gaussian learning still relies primarily on image domain features, which provide limited explicit geometric information for volumetric reasoning. Our key observation is that effective Gaussian occupancy modeling requires not only sparse primitives, but also richer geometric and semantic learning cues. In this paper, we propose VGOcc, which learns visual and geometric cues from foundation models for Gaussian modeling. VGOcc incorporates these cues into primitive initialization and refinement, yielding a representation termed Visual-Geometric Gaussians tailored to semantic occupancy prediction. Specifically, we propose Visual-Geometric Gaussian Birth to form spatially balanced Gaussian centers from ray depth hypotheses, while visual semantic features initialize primitive attributes. Next, we design Pose-Aware Feature Learning to combine foundation tokens with camera embeddings and calibrated ray information. Features from neighboring views are then aggregated at projected 3D locations for each Gaussian refinement stage. Finally, Gaussian decoder refines birth Gaussians with pose-aware features and renders them into semantic occupancy. Experiments on nuScenes demonstrate that VGOcc achieves state-of-the-art performance in vision-only 3D occupancy prediction. Codes will be available at https://github.com/JHLin42in/VGOcc.
