Search papers, labs, and topics across Lattice.
This paper introduces REGRIND, a minimalist retargeting-guided reinforcement learning pipeline that enables robots to learn dexterous manipulation from a single human demonstration. By retargeting human hand-object motion to robot references while preserving spatial and contact relationships, the method trains a residual RL policy in simulation and achieves zero-shot transfer to hardware. The resulting policies exhibit fluid, human-like manipulation across various contact-rich tasks, providing insights into effective sim-to-real transfer for dexterous manipulation.
REGRIND enables robots to master complex dexterous tasks from just one human demonstration, achieving impressive sim-to-real transfer without extensive retraining.
Recent work in humanoid whole-body control has found success with a simple recipe: retarget human motion to robot kinematic references, then train policies via reinforcement learning (RL) to track them. But how does this recipe transfer to dexterous manipulation? The answer is not obvious, as manipulation involves complex, contact-rich dynamics and requires delicate regulation of contact modes and forces. We present REGRIND, a minimalist retargeting-guided RL pipeline that learns dexterous manipulation policies from a single human demonstration. REGRIND retargets human hand-object motion to a robot reference that preserves hand-object spatial and contact relationships, trains a residual RL policy in simulation to track object-centric keypoints along that reference, and transfers the resulting policy zero-shot to hardware with careful system identification. The resulting policies produce fluid, human-like behavior on two different multi-fingered hands across contact-rich tool-use tasks, including operating a pair of scissors and turning a screwdriver. Through systematic hardware experiments, we identify and analyze the key factors that govern sim-to-real transfer in dexterous manipulation, offering practical guidance for retargeting-based learning in contact-rich settings. Videos and code are available at https://yunhaifeng.com/REGRIND.
Amazon Science