Search papers, labs, and topics across Lattice.
This paper introduces BIFROST, a novel approach for sim2real transfer that leverages a shared history encoder to exploit the inherent structure between simulation and real-world tasks. By employing a cross-domain bisimulation objective, BIFROST maps observation-action sequences that yield equivalent long-term outcomes to nearby latent states, enabling zero-shot transfer of policies from simulation to reality. Empirical results demonstrate that BIFROST outperforms traditional domain adaptation methods in both visual navigation and contact-rich manipulation tasks, highlighting its effectiveness in bridging the gap between simulated and real environments.
Zero-shot transfer of robot policies from simulation to reality is achieved by exploiting shared structure in observation-action sequences, outperforming conventional adaptation techniques.
Sim2real transfer for robot policy learning suffers due to mismatch between simulation and reality. Existing methods typically address each gap in isolation through separate adaptation modules, which are composed or layered when both gaps coexist. Yet the basis for attempting sim2real in the first place is that there is shared structure between a task in simulation and reality, where equivalent actions from equivalent configurations produce equivalent long term outcomes regardless of domain specific differences in rendering or physics. In this paper, we study whether we can identify and exploit this shared structure from raw observations to train a policy that enables zero shot transfer. We introduce BIFROST, which learns a shared history encoder on paired cross-domain data via cross-domain bisimulation objective: observation-action sequences leading to equivalent long-term behavior are mapped to nearby latent states, regardless of domain. Policies trained on these latent states in simulation transfer zero-shot to reality. We provide empirical evidence on sim2sim visual navigation and sim2real contact rich manipulation task and visual servoing task that BIFROST achieves effective transfer where domain adaptation and co-training baselines fail under both visual and dynamics domain gaps.
