Search papers, labs, and topics across Lattice.
The paper introduces WARPED, a framework for training visuomotor policies from egocentric human demonstrations using only monocular RGB data by synthesizing realistic wrist-view observations. It leverages vision foundation models to initialize the interactive scene, tracks hand-object interactions, retargets trajectories to a robotic end-effector, and synthesizes photo-realistic wrist-view observations via Gaussian Splatting. Results show that policies trained with WARPED achieve comparable success rates to those trained on teleoperated data, while significantly reducing data collection time (5-8x).
Train robots to mimic human manipulation skills using only egocentric video and achieve performance on par with teleoperation, slashing data collection time by up to 8x.
Recent advancements in learning from human demonstration have shown promising results in addressing the scalability and high cost of data collection required to train robust visuomotor policies. However, existing approaches are often constrained by a reliance on multiview camera setups, depth sensors, or custom hardware and are typically limited to policy execution from third-person or egocentric cameras. In this paper, we present WARPED, a framework designed to synthesize realistic wrist-view observations from human demonstration videos to facilitate the training of visuomotor policies using only monocular RGB data. With data collected from an egocentric RGB camera, our system leverages vision foundation models to initialize the interactive scene. A hand-object interaction pipeline is then employed to track the hand and manipulated object and retarget the trajectories to a robotic end-effector. Lastly, photo-realistic wrist-view observations are synthesized via Gaussian Splatting to directly train a robotic policy. We demonstrate that WARPED achieves success rates comparable to policies trained on teleoperated demonstration data for five tabletop manipulation tasks, while requiring 5-8x less data collection time.
