Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
This paper introduces TDV, a trajectory-based diffusion framework for robot manipulation that extracts object 6D poses from human videos to generate task-relevant trajectories, bridging the visual domain gap without explicit action labels. By employing plug-and-play pose estimation and a modality-agnostic intermediate representation, TDV mitigates background noise and viewpoint variations, enabling efficient data generation for diverse embodiments. The method achieves near 100% success rates on RLBench simulation tasks and demonstrates cross-embodiment generalization from one-shot human demonstrations in real-world experiments.
Robots can now learn complex manipulation from single human videos by extracting object trajectories, bypassing the need for teleoperation and achieving near-perfect task success.
Imitation learning enables robots to perform complex tasks by leveraging human-collected demonstrations. However, most existing approaches require labor-intensive data collection with additional teleoperation systems, which hinders scalability. Human videos represent a natural source of data that implicitly contain knowledge of manipulation behaviors. However, transferring them to robots remains challenging due to the visual domain gap and the absence of explicit action labels. In this work, we introduce TDV, a trajectory-based framework that leverages data generation to enable generalizable manipulation. TDV focuses on task-relevant objects by leveraging plug-and-play 6D pose estimation techniques to extract object trajectories from human videos, thereby mitigating the impact of irrelevant backgrounds and varying viewpoints. As a modality-agnostic intermediate representation, trajectories are inherently independent of the visual domains and different embodiments, enabling efficient data generation. Our method outperforms previous approaches on RLBench simulation tasks, achieving near 100\% success rates across seven tasks. In real-world experiments, TDV enables cross-embodiment generalization manipulation from one-shot human demonstrations by leveraging efficient trajectory generation techniques.
Tsinghua AI