Search papers, labs, and topics across Lattice.
The paper introduces OGScene3D, a novel system for incremental open-vocabulary 3D scene understanding that constructs scene graphs from partial observations. It uses a confidence-based Gaussian semantic representation coupled with hierarchical 3D semantic optimization to achieve semantic consistency and long-term global optimization using temporal memory. Experiments on benchmark datasets and real-world scenes demonstrate OGScene3D's effectiveness in open-vocabulary scene understanding by dynamically creating and updating nodes and semantic relationships in 3D scene graphs.
Robots can now build and understand 3D scene graphs on the fly, thanks to a new system that incrementally maps semantics with Gaussian confidence.
Open-vocabulary scene understanding is crucial for robotic applications, enabling robots to comprehend complex 3D environmental contexts and supporting various downstream tasks such as navigation and manipulation. However, existing methods require pre-built complete 3D semantic maps to construct scene graphs for scene understanding, which limits their applicability in robotic scenarios where environments are explored incrementally. To address this challenge, we propose OGScene3D, an open-vocabulary scene understanding system that achieves accurate 3D semantic mapping and scene graph construction incrementally. Our system employs a confidence-based Gaussian semantic representation that jointly models semantic predictions and their reliability, enabling robust scene modeling. Building on this representation, we introduce a hierarchical 3D semantic optimization strategy that achieves semantic consistency through local correspondence establishment and global refinement, thereby constructing globally consistent semantic maps. Moreover, we design a long-term global optimization method that leverages temporal memory of historical observations to enhance semantic predictions. By integrating 2D-3D semantic consistency with Gaussian rendering contribution, this method continuously refines the semantic understanding of the entire scene. Furthermore, we develop a progressive graph construction approach that dynamically creates and updates both nodes and semantic relationships, allowing continuous updating of the 3D scene graphs. Extensive experiments on widely used datasets and real-world scenes demonstrate the effectiveness of our OGScene3D on open-vocabulary scene understanding.
