Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Sim2Real via Bottlenecked Latent Reconstruction (SBLR), a novel framework that eliminates the need for sensor-specific tactile simulation by leveraging an oracle sensor for policy training and aligning real sensor embeddings at inference. The method employs a two-stage training process where policies first learn from oracle sensor latents, followed by a reconstruction phase to adapt to real sensor information loss. Experimental results on contact-rich tasks reveal that SBLR achieves performance comparable to direct tactile simulation, with hardware tests demonstrating a remarkable 85-97.5% zero-shot success rate, significantly outperforming traditional physics-based simulation approaches.
Achieving up to 97.5% success in real-world tactile tasks without any sensor-specific modeling could revolutionize how robots adapt to diverse tactile environments.
Robot sensor designs, particularly tactile sensors, are highly diverse and evolve rapidly. Modeling each sensor in simulation demands substantial domain expertise and computational approximations can degrade the fidelity of the simulated signals. We propose Sim2Real via Bottlenecked Latent Reconstruction (SBLR), a framework that avoids sensor-specific simulation entirely by (1) training policies on a simulator-native oracle sensor that is easy to construct without modeling any particular sensor (e.g. we use a point-cloud and finger-tip forces as a tactile oracle), and (2) aligning real sensor latent embeddings to those of the oracle sensor at inference time. Policy training proceeds in two-stage: the policy first learns from the oracle sensor latents, then a bottlenecked latent reconstruction adapts it to the information loss expected when using the real sensor instead of the oracle. The alignment between oracle and real sensor is learned from unpaired random-play data collected in both simulation and the real world, using rectified-flow-based transformation networks trained on nearest-neighbor pseudo-pairs. Simulation experiments on three contact-rich tasks show that SBLR matches or approaches the performance of an oracle with direct access to tactile simulation. Hardware experiments on Peg Insertion and Gear Meshing with GelSight Mini and DIGIT sensors demonstrate 85-97.5% zero-shot success without requiring any sensor-specific modeling or calibration, outperforming a physics-based tactile simulation baseline by 7.5-15%.