Search papers, labs, and topics across Lattice.
This paper introduces Simulation Pre-training for Dexterity (SPD), a novel framework that leverages simulated data collected through human manipulation in a VR environment to enhance the data efficiency of training dexterous robotic hands. By collecting 75 hours of multi-task manipulation data, SPD enables effective pre-training of a causal transformer model, which is then fine-tuned with minimal real-world demonstrations. The results indicate that SPD significantly outperforms traditional behavior cloning methods, demonstrating the potential of simulation-based pre-training for advancing real-world dexterous manipulation tasks.
Simulation-based pre-training can drastically improve the dexterity of robotic hands, outperforming traditional training methods with just a fraction of real-world data.
Large-scale pre-training has made robot policy fine-tuning increasingly data-efficient, but this progress has largely been driven by datasets and embodiments built around simple parallel-jaw grippers. Dexterous, multi-fingered hands remain comparatively data-starved because real teleoperation is costly to scale, while human hand video is off-embodiment and requires lossy pose estimation and retargeting. We introduce Simulation Pre-training for Dexterity (SPD), a pre-training framework for dexterous manipulation that uses data entirely collected in simulation. In SPD, humans manipulate virtual objects inside a VR headset, enabling on-embodiment trajectories and robot-free collection. With the help of five operators, we collect 75 hours of multi-task dexterous manipulation over one week, and use it to pre-train a causal transformer on a sequence modeling objective. We study the benefits of simulation pre-training on real-world tasks by fine-tuning on 1-2 hours of physical demonstrations on a 56-DoF bimanual dexterous setup. We find that our approach outperforms training behavior cloning policies from scratch, showing that simulation teleoperation is a viable pre-training source for real-world dexterous manipulation. We perform ablation studies, measuring the benefits of history conditioning and short action chunks for reactive control.
Stanford HAI