Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces "World Translation," a novel approach that addresses the sim-to-real gap in robotic policy deployment by extracting unobservable dynamics information backward from observed transitions. By leveraging the deterministic nature of simulators alongside the accuracy of learned dynamics models, the method translates this information across domains without requiring informative observation histories. Experimental results demonstrate significant improvements in dynamics modeling accuracy and successful policy transfer in real-robot deployments, particularly in scenarios where unobservable factors hinder traditional methods.
Unobservable dynamics can be extracted backward to enhance robotic policy transfer, significantly narrowing the sim-to-real gap even in challenging scenarios.
The gap between simulation and reality remains a fundamental challenge in deploying simulation-trained robotic policies in the real world. Real-to-sim methods narrow this gap from the real side, learning transition dynamics from real data to build a more realistic digital world. Learned dynamics models are their dominant instance. Such methods, however, face a partial observability problem: the same observation may branch to different transitions due to unobservable factors. Existing methods assume these factors can be recovered from observation history. However, this may fail whenever observation history is uninformative, such as a sudden contact event with no prior warning. To address this limitation, we propose \textit{World Translation}, which exploits a complementary strength of simulators and learned dynamics. Simulators are deterministic but physically imperfect, while learned models are accurate but underdetermined under partial observability. Rather than predicting transitions forward from history, we extract the unobservable dynamics information backward from an observed transition, then translate this feature across simulation and reality as an unpaired domain-translation problem that preserves dynamics content while transferring domain style. Experiments across humanoid, quadruped, and manipulator platforms show that our method achieves more accurate dynamics modeling than baselines, with the largest gains when unobservable factors cannot be recovered from observation history. Real-robot deployment on Go2 quadruped confirms improved policy transfer.