Search papers, labs, and topics across Lattice.
This paper introduces Pegasus, a framework that translates human manipulation videos into robot-learnable data by bridging the embodiment gap between human morphology and robot hardware. By constructing a graph-based intermediate representation, Pegasus enables robots to learn from existing human demonstrations without the need for extensive hardware-specific data collection. Evaluation across various benchmarks shows that this approach allows for reliable cross-embodiment translation, effectively reframing robot data generation as a scalable knowledge transfer problem.
Robots can now learn from billions of human manipulation videos without needing custom hardware, thanks to a novel graph-based translation framework.
The key bottleneck in embodied AI is not model architecture but data. Although billions of human manipulation videos exist online, robots cannot directly learn from them due to the embodiment gap between human morphology and robot hardware. We introduce Pegasus, a low-resource framework that bridges this gap by translating human demonstrations into robot-learnable data through structured knowledge transfer. Instead of relying on raw video prompts, Pegasus constructs a graph-based intermediate representation: a Task Graph extracted from human videos is transformed through Affordance and Constraint Graphs into a Robot Planning Graph for robot-conditioned video generation. A hierarchical affordance latent space models the relationship between object states, affordances, and tasks, enabling generalization beyond object identities. A closed-loop physics verifier further filters invalid generations using kinematic feasibility, collision constraints, and joint limits. We evaluate Pegasus across a range of egocentric manipulation benchmarks, including GTEA Gaze+ and EPIC-KITCHENS-100, and diverse robot embodiments, assessing Task Correctness, Executability, State Consistency, and Learnability. Results demonstrate reliable cross-embodiment translation and show that robot data generation can be reframed from a hardware collection problem into a scalable, low-resource knowledge transfer problem.
