Search papers, labs, and topics across Lattice.
DGSG-Mind is introduced, a novel hybrid 3D scene representation that combines probabilistic voxel grids with explicit 3D Gaussians for robust cross-modal instance fusion and incremental semantic mapping in dynamic environments. The system constructs a hierarchical scene graph on top of the instance Gaussian map and introduces a "3D Gaussian Mind" for multimodal reasoning using structural relations, spatial-semantic information, and visually annotated RoI Gaussian renderings. Experiments demonstrate state-of-the-art zero-shot 3DVG performance on self-reconstructed maps, strong performance in 3D open-vocabulary semantic segmentation and scene reconstruction, and successful deployment on real-world robots for target-oriented reasoning.
Robots can now understand and interact with changing environments far more effectively thanks to a new 3D scene graph representation that fuses visual and semantic information with Gaussian representations.
Integrating open-vocabulary semantic information into dynamic 3D scene representations is essential for long-term embodied scene understanding. However, existing methods often suffer from fragile instance association due to incomplete cross-view cues, while their limited ability to handle object-level topological changes restricts long-term robotic task execution. Moreover, current 3D scene understanding methods either rely on simple feature matching without explicit spatial reasoning or assume offline ground-truth 3D geometry. To address these challenges, we present DGSG-Mind, a hybrid instance-aware 3D Gaussian dynamic scene graph system with an embodied reasoning agent. Our system couples a probabilistic voxel grid with explicit 3D Gaussians to enable robust cross-modal instance fusion and incremental semantic mapping. It handles dynamic changes through Gaussian-based visual relocalization and localized masked refinement guided by geometric-semantic consistency. Built on the instance Gaussian map, DGSG-Mind further constructs a hierarchical scene graph and develops the 3D Gaussian Mind, which integrates structural relations, spatial-semantic information, and visually annotated RoI Gaussian renderings for multimodal reasoning. Extensive experiments show that DGSG-Mind achieves the best zero-shot 3DVG performance among methods operating on self-reconstructed maps, while also delivering strong performance in 3D open-vocabulary semantic segmentation and scene reconstruction. We further deploy DGSG-Mind on real-world robots to demonstrate its target-oriented reasoning and dynamic update capabilities. The project page of DGSG-Mind is available at https://icr-lab.github.io/DGSG-Mind
