Search papers, labs, and topics across Lattice.
This paper introduces LaST-HD, a novel paradigm for robot manipulation that leverages human-hand demonstrations to align physical dynamics between human and robot actions in a shared latent reasoning space. By training an auxiliary action-conditioned world model on unpaired trajectories, LaST-HD synthesizes unified latent targets that enhance the robot's ability to internalize shared physical dynamics, leading to improved generalization and adaptation in novel environments. The approach achieves over 90% accuracy with minimal human data input, demonstrating its efficiency and scalability in real-world applications.
LaST-HD achieves over 90% accuracy in robot manipulation tasks using just 20 minutes of low-cost human demonstration data, revolutionizing how robots learn from human actions.
Human-hand demonstrations provide a direct and scalable source of physical interaction data for robot learning. While manual retargeting is indispensable for establishing kinematic action correspondence across different morphologies, robust transfer requires going beyond geometry to address the underlying alignment of physical dynamics between human and robot manipulation. To address this, we introduce LaST-HD, a novel human-to-robot action learning paradigm that extends reasoning-before-acting VLA by aligning human-hand and robot demonstrations in a shared latent reasoning space. Rather than mimicking human kinematics, LaST-HD trains an auxiliary action-conditioned world model on unpaired human-hand and robot trajectories to synthesize unified latent targets. After aligning cross-embodiment representations in this shared forward-dynamics space, these targets supervise LaST-HD's latent reasoning process, enabling it to internalize shared physical dynamics and drive efficient human-hand action learning. Moreover, we develop Out-of-Lab (OOL) Glove, a low-cost motion-capture glove tailored to LaST-HD for human-hand data collection. The captured human data provide precise keypoints and serve as universal action supervision across grippers and dexterous hands. Armed with the aligned latent space and high-fidelity human-hand data, we develop a progressive mixed-to-human training recipe comprising mixed human-robot co-training and human-hand online correction post-training. Through mixed co-training, LaST-HD improves generalization to novel objects, scenes, and positions using only human-hand demonstrations. With online correction, LaST-HD further adapts to novel environments and achieves over 90\% accuracy using only 20 minutes of OOL glove data.
Microsoft Research