Search papers, labs, and topics across Lattice.
This paper introduces RoboSnap, a framework that transforms a single RGB image into a simulation-ready scene, facilitating generalizable robot learning and reproducible policy evaluation. By employing a layered design that distinguishes between physics-critical interaction areas and visual context, RoboSnap achieves reliable trajectory replay and task-specific synthetic data generation. Experiments demonstrate its effectiveness in creating stable environments for robot interaction and enhancing sim-real correlations, supported by the DROID-Sim dataset of 564 real-world scenes.
Transforming a single image into a robust simulation environment could revolutionize how robots learn and evaluate policies in real-world scenarios.
Recovering real-world scenes as interactive simulation environments can enable generalizable robot learning and reproducible policy evaluation. However, constructing scenes that are both physically stable and visually faithful remains slow and expensive. In this work, we present RoboSnap, a real-to-sim framework that turns a single RGB image into a simulation-ready scene. The key idea is a layered design that separates the physics-critical interaction area from the surrounding visual context: collision-aware foreground assets are refined for stable robot interaction, while a 3D Gaussian splatting visual layer preserves faithful background appearance under novel views. Experiments on DROID scenes and real-robot tasks show that RoboSnap achieves reliable trajectory replay in the recovered scenes, supports task-specific synthetic data generation for policy training, and yields meaningful sim-real correlation for policy evaluation. To further support real-to-sim research, we introduce DROID-Sim, a real-to-sim companion dataset constructed from 564 real-world scenes in DROID. Extensive experiments suggest that the value of real-to-sim methods lies not only in high-fidelity visual reconstruction, but in turning real environments into reusable infrastructure for robot learning and evaluation.
