Search papers, labs, and topics across Lattice.
This study investigates the transferability of world-action models trained on synthetic data to real-world robotic manipulation tasks, addressing a significant challenge in sim-to-real learning. By leveraging a video diffusion model called Cosmos Policy and employing extensive domain randomization, the authors successfully trained a model using approximately 800 synthetic demonstrations per task without any real-world data. The resulting policy achieved a 35% average success rate on a Franka Robot, marking a significant milestone in the deployment of learned manipulation policies in real environments.
Achieving a 35% success rate in real-world robotic tasks using only synthetic data marks a groundbreaking advance in sim-to-real transfer for world-action models.
Bridging the sim-to-real gap is a core challenge in deploying learned manipulation policies. Sim-to-real learning is attractive because it can replace expensive real robot demonstrations with scalable synthetic data, yet world-action models have not previously been shown to transfer from simulation to real robotic manipulation. We study whether a world-action model can be trained from synthetic priors and deployed zero-shot in the real world. To this end, we build upon Cosmos Policy, a video diffusion model adapted for visuomotor control. We construct simulation environments with extensive domain randomization and generate demonstrations using the AnyTask motion planning pipeline. We evaluate our approach across object lifting, drawer opening, and pick-and-place tasks using ${\sim}800$ synthetic demonstrations per task and no real demonstrations. When deployed zero-shot on a Franka Robot, our policy attains a 35\% average success rate. To our knowledge, this represents the first successful sim-to-real transfer of a world-action model for robotic manipulation.
