Search papers, labs, and topics across Lattice.
HandelBot addresses the challenge of transferring dexterous manipulation policies from simulation to real-world piano playing by using a two-stage adaptation pipeline. The approach first refines spatial alignments through structured adjustments of finger joints based on physical rollouts, and then employs residual reinforcement learning for fine-grained corrective actions. Experiments demonstrate HandelBot's ability to perform precise bimanual piano playing, outperforming direct simulation deployment by 1.8x while requiring only 30 minutes of physical interaction data.
A robot can now play recognizable piano songs after just 30 minutes of real-world training, closing the sim-to-real gap for high-precision bimanual manipulation.
Mastering dexterous manipulation with multi-fingered hands has been a grand challenge in robotics for decades. Despite its potential, the difficulty of collecting high-quality data remains a primary bottleneck for high-precision tasks. While reinforcement learning and simulation-to-real-world transfer offer a promising alternative, the transferred policies often fail for tasks demanding millimeter-scale precision, such as bimanual piano playing. In this work, we introduce HandelBot, a framework that combines a simulation policy and rapid adaptation through a two-stage pipeline. Starting from a simulation-trained policy, we first apply a structured refinement stage to correct spatial alignments by adjusting lateral finger joints based on physical rollouts. Next, we use residual reinforcement learning to autonomously learn fine-grained corrective actions. Through extensive hardware experiments across five recognized songs, we demonstrate that HandelBot can successfully perform precise bimanual piano playing. Our system outperforms direct simulation deployment by a factor of 1.8x and requires only 30 minutes of physical interaction data.
Stanford HAI