Search papers, labs, and topics across Lattice.
DeWorldSG introduces a framework for generating robust 3D Semantic Scene Graphs from RGB-D sequences by leveraging depth-guided filtering to estimate instance-level geometric 3D Gaussian distributions. This approach overcomes the limitations of existing methods that suffer from unstable representations and relational sparsity due to frame-wise inference by aggregating spatiotemporal evidence and refining relations with contextual priors from a world model. Experimental results on the 3DSSG and ReplicaSSG datasets show that DeWorldSG achieves state-of-the-art performance, significantly enhancing triplet and predicate recall, which is crucial for applications in robotic manipulation and augmented reality.
Triplet recall improves by 77.4% with DeWorldSG, setting a new benchmark for 3D scene graph generation.
We present DeWorldSG, a novel framework that generates spatio-temporally robust 3D Semantic Scene Graphs from RGB-D sequences. Existing methods often struggle to construct reliable 3D scene graphs due to unstable 3D object representations and missing relations caused by frame-wise inference. DeWorldSG addresses these issues by estimating instance-level geometric 3D Gaussian distributions through depth-guided filtering and representing each object as a probabilistic 3D node rather than a single projected point. To mitigate relational sparsity from frame-wise inference, our framework further aggregates spatiotemporal evidence across object pairs and refines relations using contextual priors derived from a world model (V-JEPA 2). Experiments on the 3DSSG and ReplicaSSG datasets demonstrate state-of-the-art (SoTA) performance in both object and predicate prediction, while producing temporally consistent scene structures. In particular, our method improves triplet recall by 77.4% and predicate recall by 23.2% over prior SoTA approaches, making it suitable for robotic manipulation and AR applications. Our code and models are open-sourced.
