Search papers, labs, and topics across Lattice.
The paper introduces Ego2Robot, a scalable pipeline that synthesizes robot training data from egocentric human manipulation videos through action retargeting and robot-arm visual synthesis. This approach generates 18,561 hours of diverse training data across 15 robot morphologies, significantly enhancing the scale and variety of datasets available for training robot manipulation policies. Experiments demonstrate that joint pretraining using the synthesized data improves out-of-distribution generalization, validated through real-robot deployment, highlighting the effectiveness of this method in advancing robot learning capabilities.
Joint pretraining on Ego2Robot-synthesized data boosts robot generalization, achieving unprecedented scale and diversity in training datasets.
Learning generalizable robot manipulation policies requires large-scale and diverse demonstration data. Egocentric human manipulation videos offer rich scene and task diversity, and prior work has shown that retargeting and rendering such videos into robot-format data can yield effective per-task policies at small scale. However, whether this approach can provide pretraining benefits for vision-language-action models at scale remains unexplored. We present \textbf{Ego2Robot}, a scalable pipeline that converts egocentric human manipulation videos into robot training data through action retargeting, robot-arm visual synthesis, and multi-level quality curation. Ego2Robot supports both curated datasets and in-the-wild videos, producing 18,561 hours of robot training data spanning 15 robot morphologies, making it the largest ego-to-robot dataset to date. To evaluate generalization, we extend RoboTwin2.0 with disentangled perturbation axes covering visual appearance, scene layout, embodiment morphology, and task semantics. Experiments show that joint pretraining on Ego2Robot-synthesized and robot data consistently improves out-of-distribution generalization across multiple perturbation types, with benefits validated on real-robot deployment. Project page: https://www-ye.github.io/ego2robot_blog/
