Search papers, labs, and topics across Lattice.
This paper introduces Semantic Radiance Fields (SRFs) as a novel approach to create simulators that effectively combine geometric fidelity with semantic richness for training spatial reasoning agents. By transforming 2D semantic segmentations into a 3D radiance field, SRFs provide a unified representation that supports realistic rendering and semantic queries, thereby addressing the limitations of existing synthetic and reconstructed simulators. The authors demonstrate the utility of SRFs through an application in an orchard apple-reaching task, showcasing how this method can enhance the training and evaluation of embodied agents in diverse real-world environments.
SRFs enable the creation of simulators that blend realistic scene appearance with precise semantic information, revolutionizing spatial reasoning training for embodied agents.
Training and evaluating spatial reasoning in embodied agents requires diverse environments that are both geometrically faithful and semantically queryable. Synthetic simulators offer ground truth semantics but sacrifice realism; simulators based on reconstructions of real-world environments have realistic appearance but lack ground truth semantics by default. We propose using Semantic Radiance Fields (SRF) as simulators for spatial reasoning agents. SRFs are a representation that unifies these requirements by lifting 2D semantic segmentations from pretrained vision models into a 3D radiance field that jointly encodes geometry, appearance, and per-class semantic identity. The resulting fields are reconstructed from posed RGB captures of real scenes and support novel-view synthesis, semantic and free-space queries within a single grounded representation. This enables the efficient generation of diverse real-world environments to train and evaluate spatial reasoning models. As an example application, we outline an SRF-driven simulator for an orchard apple-reaching task, in which the radiance field supplies camera rendering, semantic ground truth, and occupancy queries to a physics engine.
