Search papers, labs, and topics across Lattice.
This paper introduces Image2Sim, a neural simulation framework that generates high-quality interactive environments from RGB-D image sequences, addressing the limitations of both real-world datasets and traditional synthetic simulators. By decoupling 3D spatial anchoring from photorealistic observation synthesis, Image2Sim constructs nearly 20,000 interactive scenes and synthesizes over 10 million navigation training samples. The navigation models trained in these environments demonstrate significant performance improvements on major benchmarks and effectively transfer to real-world scenarios, highlighting the potential of scalable neural simulation for embodied navigation tasks.
Navigation models trained in Image2Sim's synthetic environments outperform traditional methods, achieving zero-shot transfer to real-world settings.
Embodied navigation aims to build agents that interpret multimodal goals, reason in 3D space, and reach target destinations reliably in the real world. However, progress remains constrained by the lack of scalable, high-fidelity, and physically grounded interactive environments. Although real-world scanned datasets offer visual realism, they are limited by scale. In contrast, synthetic simulators scale more easily but often exhibit large sim-to-real gaps. We introduce Image2Sim, a real-time neural simulation framework that constructs high-quality interactive environments from posed RGB-D image sequences. The central idea is to decouple 3D spatial anchoring from photorealistic observation synthesis. For scene construction, Image2Sim uses a feed-forward feature Gaussian model that lifts posed RGB-D observations into a 3D feature-Gaussian representation in a single pass. For rendering, we propose a Geometry-Aware One-Step Pixel Flow model that transforms sparse and noisy Gaussian projections into high-quality panoramic RGB-D observations. Image2Sim also serves as a fully automated embodied data engine that generates high-fidelity observations, executable actions, and diverse navigation instructions at scale. It converts large collections of videos and images into nearly 20K interactive scenes and synthesizes more than 10 million navigation training samples. Navigation models trained entirely in these neural environments achieve strong improvements on major benchmarks and transfer effectively to real-world zero-shot settings. These results suggest that scalable neural simulation can serve as a practical training substrate for embodied navigation at scale.
NUS