Search papers, labs, and topics across Lattice.
The paper introduces MoMa-SG, a framework for constructing semantic-kinematic 3D scene graphs that represent articulated objects and their relationships in dynamic environments. It segments object interactions from RGB-D sequences, tracks object motion robustly to occlusions, and estimates articulation models using a unified twist estimation formulation for revolute and prismatic joints. The framework also associates objects with articulations, detects contained objects, and is evaluated on the new Arti4D-Semantic dataset and in real-world robotic manipulation experiments.
Robots can now build 3D scene graphs that understand how objects move, enabling more robust manipulation of articulated objects in real-world environments.
Semantics has enabled 3D scene understanding and affordance-driven object interaction. However, robots operating in real-world environments face a critical limitation: they cannot anticipate how objects move. Long-horizon mobile manipulation requires closing the gap between semantics, geometry, and kinematics. In this work, we present MoMa-SG, a novel framework for building semantic-kinematic 3D scene graphs of articulated scenes containing a myriad of interactable objects. Given RGB-D sequences containing multiple object articulations, we temporally segment object interactions and infer object motion using occlusion-robust point tracking. We then lift point trajectories into 3D and estimate articulation models using a novel unified twist estimation formulation that robustly estimates revolute and prismatic joint parameters in a single optimization pass. Next, we associate objects with estimated articulations and detect contained objects by reasoning over parent-child relations at identified opening states. We also introduce the novel Arti4D-Semantic dataset, which uniquely combines hierarchical object semantics including parent-child relation labels with object axis annotations across 62 in-the-wild RGB-D sequences containing 600 object interactions and three distinct observation paradigms. We extensively evaluate the performance of MoMa-SG on two datasets and ablate key design choices of our approach. In addition, real-world experiments on both a quadruped and a mobile manipulator demonstrate that our semantic-kinematic scene graphs enable robust manipulation of articulated objects in everyday home environments. We provide code and data at: https://momasg.cs.uni-freiburg.de.
