Search papers, labs, and topics across Lattice.
This paper introduces V2P-Manip, a novel framework that learns dexterous manipulation policies from monocular human demonstration videos, addressing the challenge of scaling precise action sequences for autonomous robots. By integrating 3D asset acquisition, trajectory estimation, and a two-stage refinement process, the framework ensures both visual fidelity and physical plausibility in the learned policies. Evaluations on the TACO and OakInk benchmarks reveal that V2P-Manip achieves over 75% success in various synthetic manipulation tasks, outperforming previous methods in pose accuracy and adaptability to unstructured environments.
Learning dexterous manipulation from monocular videos could redefine how robots acquire complex skills without costly teleoperation data.
Achieving autonomous robotic dexterous manipulation requires precise, human-like action sequences at scale. As a scalable supplement to costly teleoperation data, extracting trajectories with both visual fidelity and physical plausibility from monocular videos represents a promising frontier in embodied AI. To this end, we introduce V2P-Manip, an efficient framework designed to learn dexterous manipulation policies directly from human demonstration videos. We establish an efficient, integrated pipeline encompassing 3D asset acquisition, trajectory estimation, and dexterous policy learning. To bridge the gap between visual perception and physical constraints, we introduce a two-stage refinement process to enforce spatial alignment and physical consistency. Evaluations on the TACO and OakInk benchmarks demonstrate that our approach significantly outperforms previous methods in pose accuracy, adaptability to unstructured environments, and training efficiency. Ultimately, experimental results confirm an average success rate of over 75% across multiple synthetic manipulation tasks and validate the adaptability of the extracted manipulation priors across diverse dexterous hand embodiments.
