Search papers, labs, and topics across Lattice.
This paper introduces RoboDream, a novel embodiment-centric world model that synthesizes photorealistic robot demonstrations in diverse environments, addressing the challenge of expensive real-world data collection. By anchoring generation to rendered robot motion and conditioning on explicit scene and object priors, the model effectively separates trajectory execution from environment synthesis, enabling innovative data scaling techniques such as retrieval and rebirth, and prop-free teleoperation. Experimental results show that RoboDream significantly enhances downstream policy performance while reducing the need for real-world data across various manipulation tasks.
By enabling robots to learn from synthesized data in novel contexts without additional motion data, RoboDream transforms the landscape of robot learning efficiency.
Scaling robot learning requires large-scale, diverse demonstrations, yet real-world data collection via teleoperation remains prohibitively expensive and time-consuming. While video diffusion models offer a promising avenue for data scaling, existing generative approaches are often limited to superficial visual augmentation, or suffer from embodiment hallucinations that yield physically infeasible motions. We present a generalizable embodiment-centric world model that achieves scalable data generation by synthesizing photorealistic demonstrations with novel objects, in novel scenes, and from novel viewpoints. Our approach anchors generation to rendered robot motion while conditioning on explicit scene and object priors, effectively decoupling trajectory execution from environment synthesis. This formulation has the potential to unlock two powerful data scaling capabilities: (1) retrieval and rebirth, which repurposes existing trajectories into entirely new contexts without new motion data; and (2) prop-free teleoperation, where operators manipulate empty air and the model hallucinates the target objects and scene afterwards, eliminating reset time. We demonstrate with real-world experiments that our generated data consistently improves downstream policy performance and significantly reduces real-world data requirements across diverse manipulation tasks.
MIT CSAIL