Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to constructing hierarchical 3D scene graph forests by leveraging foundation models to capture open semantic relationships, overcoming the limitations of rigid pre-determined relationship classes in existing methods. By utilizing a vision-language model (VLM) to identify instance-specific concept-nodes and a large language model (LLM) to infer broader relationships, the authors create a more expressive framework that enhances robotic scene understanding and task execution. Evaluations on the uHumans2 and ScanNet datasets demonstrate the effectiveness of this method, particularly in an open-vocabulary object-retrieval task, showcasing its applicability in real-world robotic deployments.
Foundation models can transform rigid 3D scene graphs into rich, semantically aware forests that enhance robotic understanding and interaction with complex environments.
Operating in complex real-world environments requires robots to understand their surroundings on a functional semantic level. This demands a detailed multi-layer world model capturing the complex relations of its surroundings. Hierarchical 3D scene graphs address this challenge by integrating geometric, semantic, and relational data within a unified spatial framework. However, current 3D scene graph approaches often restrict themselves to rigid structures of pre-determined relationship classes, mostly neglecting important semantic connections, like causal connections or environmental contexts. This paper explores the potential of foundation models to build forests of 3D scene graphs with open semantic relationships to improve scene understanding and robotic task execution. We propose a method where instance-specific concept-nodes and relationships are first identified by a VLM and extended upon by a LLM, inferring broader, more abstract concept-nodes and relationships through reasoning. These object-nodes, concept-nodes, and relationships are then assembled into a forest of hierarchical 3D scene graphs, enhanced with concept-nodes to represent abstract concepts. Evaluations were conducted on the uHumans2 and ScanNet indoor dataset, validating the accuracy and relevance of the generated relationships. Downstream suitability of scene-graph forests for robotics applications is demonstrated in an open-vocabulary object-retrieval task utilizing both ScanNet data and a real-world indoor deployment using a Boston Dynamics Spot. This paper leverages foundation models to create more expressive, semantically deep 3D hierarchical scene graphs and demonstrates their potential to advance semantic and environmental understanding in robotics.
