Search papers, labs, and topics across Lattice.
This study explores the use of large language models (LLMs) to automate the grounding of scene objects to formal ontology classes in Universal Scene Description (USD) environments, addressing the limitations of traditional, manually curated dictionaries. The researchers demonstrate that LLMs can achieve 90-96% accuracy in matching descriptive names and 49-89% with abbreviated names, significantly outperforming existing dictionary and embedding methods. Additionally, they reveal that LLMs rely heavily on semantic cues from the scene graph, with a notable drop in accuracy when these cues are anonymized.
LLMs can achieve up to 96% accuracy in grounding scene objects to ontologies without any training, revolutionizing the way we automate knowledge graph construction from 3D scenes.
Constructing knowledge graphs from 3D simulation scenes is essential for robot task reasoning, but the key bottleneck, grounding scene objects to formal ontology classes, still relies on manually curated dictionaries that are brittle and do not generalize across assets. We investigate whether large language models (LLMs) can automate this grounding step for Universal Scene Description (USD) scenes as a zero-shot, training-free alternative. On a kitchen scene (125 objects) with SOMA-HOME Ontology, LLMs achieve 90-96% exact-match accuracy with descriptive names and 49-89% with abbreviated names, substantially outperforming dictionary and embedding baselines. Under fully opaque names, context-augmented prompting recovers up to 48%. Feature ablation reveals that LLMs primarily exploit semantic cues in the scene graph (sibling names and parent paths); anonymizing these cues reduces accuracy to 0-6%, while geometry alone yields only 4-17%.
