Search papers, labs, and topics across Lattice.
This paper introduces LUCID, a two-stage framework that leverages unstructured human videos to learn task intent for robot skill acquisition, enabling scalable and embodiment-agnostic learning. By predicting short-horizon intent from observations and translating this intent into robot actions through an embodiment-specific policy, LUCID demonstrates effective zero-shot transfer across diverse manipulation tasks. Evaluated on real-world tasks like stirring and wiping, the framework shows promising results with minimal supervision, highlighting its potential for broad applicability in robotic learning.
Unstructured human videos can unlock scalable robot skill acquisition, enabling zero-shot transfer across diverse tasks with minimal supervision.
The most widely-adopted robot learning pipelines today learn skills from robot demonstrations or structured human data, which are expensive to collect and tied to specific embodiments. In contrast, unstructured human videos provide a scalable alternative. They contain diverse manipulation demonstrations across objects, scenes, and strategies, but are not directly connected to robot action. We propose LUCID, a two-stage framework that learns task intent from unstructured human videos drawn from internet-scale datasets and learns robot control in massively-parallel simulation. The intent model predicts short-horizon intent (what should happen next in the scene) from the current observation in closed loop. An embodiment-specific sensorimotor policy converts this intent into robot actions. The intent interface is shared across controllers, so the same intent model can be applied to different embodiments, from our primary dexterous hand to a parallel-jaw gripper. We evaluate LUCID on five real-world manipulation tasks: stirring, wiping, and binning supervised by only internet video, with zero-shot transfer to novel scenes and object instances; and push-T and cable routing supervised by 1 hr each of self-collected smartphone video. Project page: https://lucid-robot.github.io/.
CMU ML