Search papers, labs, and topics across Lattice.
HumanEgo learns robot manipulation policies from human egocentric video by constructing entity-level representations of hand-object interactions to bridge the human-robot embodiment gap. A flow matching policy is trained with dense auxiliary objectives to maximize the information extracted from limited human demonstrations. The resulting policies achieve 92.5% success across real-world tasks using only 30 minutes of human video, outperforming robot teleoperation and generalizing zero-shot to new robots and environments.
Just 30 minutes of human egocentric video can train robots to perform complex manipulation tasks with superhuman dexterity.
Human egocentric video captures rich manipulation demonstrations without any robot hardware, yet transferring these skills to robots remains challenging due to the embodiment gap between human and robot in both visual appearance and kinematics. We present HumanEgo, a framework that bridges the embodiment gap by lifting each human demonstration to an entity-level representation of hand-object interaction, and training a flow matching policy with dense auxiliary objectives that amplify supervision from every trajectory. HumanEgo is robot-data-free, hardware-agnostic, data-efficient, and zero-shot human-to-robot transferable. With only 30 minutes of human videos per task, HumanEgo achieves 92.5% average success across four real-world tasks (75% with just 15 minutes), outperforms matched-time robot teleoperation by 41%, and robustly transfers zero-shot across novel robots, cameras, and environments.
Amazon Science