Search papers, labs, and topics across Lattice.
This paper introduces EgoWAM, a novel framework that leverages egocentric human data to enhance robot manipulation through World Action Models (WAMs). By isolating the world representation from non-transferable factors, the authors demonstrate that using DINO and 3D motion flow as world prediction targets significantly improves generalization and performance in real-world tasks compared to traditional behavior cloning. The results show that DINO enhances out-of-distribution generalization by up to 4x, while 3D flow boosts in-domain performance by 20-30%, highlighting the effectiveness of the proposed approach.
DINO and 3D motion flow can quadruple generalization capabilities for robots trained with egocentric human data, far surpassing traditional methods.
Egocentric human data offers scalable supervision for robot manipulation. However, behavior cloning entangles transferable content like objects, scenes, and task semantics, with non-transferable factors like human morphology, head motion, and behavioral style. We study whether World Action Models (WAMs) provide a better training signal by requiring policies to predict not only actions, but also how the scene evolves. The central question is what world representation best enables human-to-robot transfer. We hypothesize that an effective world target should abstract appearance, capture agent-invariant physical effects, and separate camera motion from environment change. We introduce EgoWAM, a controlled human-robot co-training framework that fixes the policy backbone, action head, and data mixture while varying only the world prediction target, comparing Pixel, DINO, and 3D motion flow. Across three real-world bimanual tasks, WAM co-training scales more effectively with in-the-wild egocentric human data than behavior cloning. Pixel-based prediction transfers weakly, while DINO and 3D flow yield substantial gains: DINO improves out-of-distribution object and scene generalization by up to 4x, and 3D flow improves in-domain performance by 20-30%. More details: https://gatech-rl2.github.io/egowam.github.io
