Search papers, labs, and topics across Lattice.
D3D-GEN is a novel system for generating interactive 3D worlds tailored for social robotics, addressing the challenge of balancing realism and simulability in simulation environments. By integrating a domain agent with a retrieval-augmented generation (RAG) pipeline, the system automates the collection of domain-specific knowledge and the creation of realistic floorplans and object placements without relying on fixed 3D model databases. The result is a versatile tool that enables rapid generation of diverse, interactive environments for various domains, enhancing the training and validation of Embodied AI in social navigation tasks.
Generate fully interactive 3D worlds on-the-fly, tailored to specific domains, without the need for pre-existing 3D models.
Training and validation of Embodied AI for social navigation critically depends on realistic simulation environments, yet many current approaches fail to find a balance between realism and simulability. We propose D3D-GEN, a novel world generation system that combines a domain agent with a retrieval-augmented generation (RAG) pipeline grounded in that domain. Our system enables users to rapidly generate domain-grounded, fully interactive 3D worlds by automating both the collection of domain knowledge and the synthesis of realistic floorplans and object placements, without dependence on any fixed 3D model database. Given a domain description prompt, the research agent collects publicly accessible domain-specific data and constructs a persistent domain database. Using this database, our RAG pipeline generates plausible floorplans and object placements by dynamically querying a user-provided semantic database, which can be easily extended or modified. The output is a fully interactive 3D world loadable by the popular simulators Isaac Sim and Gazebo. With our approach, we have built databases for several common domains (indoor residential, hospital, office) and generated dozens of distinct, plausible simulation environments for each domain. We present D3D-GEN with a local web frontend that facilitates rapid, interactive world generation for robot simulation.
