Search papers, labs, and topics across Lattice.
This paper introduces a novel 3D scene understanding approach that unifies appearance, semantics, and geometry by leveraging language and geometry grounded sparse voxel representations. The method employs sparse voxels as primitives, using appearance, density, feature, and confidence fields to represent the 3D scene holistically, and introduces a feature modulation module to distill language features from a 2D foundation model. By integrating geometric distillation via depth correlation and pattern consistency regularization, the model effectively transfers geometric knowledge from a geometry foundation model into the 3D scene representations, leading to improved holistic scene understanding and reconstruction.
By unifying appearance, semantics, and geometry within sparse voxel representations, this work achieves state-of-the-art holistic 3D scene understanding and reconstruction.
Existing 3D open-vocabulary scene understanding methods mostly emphasize distilling language features from 2D foundation models into 3D feature fields, but largely overlook the synergy among scene appearance, semantics, and geometry. As a result, scene understanding often deviates from the underlying geometric structure of scenes and becomes decoupled from the reconstruction process. In this work, we propose a novel approach that leverages language and geometry grounded sparse voxel representations to comprehensively model appearance, semantics, and geometry within a unified framework. Specifically, we use 3D sparse voxels as primitives and employ an appearance field, a density field, a feature field, and a confidence field to holistically represent a 3D scene. To promote synergy among the appearance, density, and feature fields, we construct a feature modulation module and distill language features from a 2D foundation model into our 3D scene model. In addition, we integrate geometric distillation into feature field distillation to transfer geometric knowledge from a geometry foundation model to our 3D scene representations via depth correlation regularization and pattern consistency regularization. These components work together to synergistically model the appearance, semantics, and geometry of the 3D scene within a unified framework. Extensive experiments demonstrate that our approach achieves superior overall performance compared with state-of-the-art methods in holistic scene understanding and reconstruction.
