Search papers, labs, and topics across Lattice.
This paper introduces the Dexterous Point Policy framework, which learns dexterous manipulation policies directly from human demonstration videos, eliminating the need for costly robot-specific data collection. By leveraging a unified 3D keypoint representation for both observations and actions, the approach effectively bridges the embodiment gap between human and robotic manipulation. The framework achieves a remarkable 75.0% success rate on real-robot tasks, significantly outperforming a state-of-the-art baseline that only reached 1.0%, while also demonstrating strong generalization to novel scenarios.
Learning dexterous manipulation policies from human videos can achieve a 75% success rate on real-robot tasks without any robot-specific demonstrations.
Robotic foundation models pre-trained on human demonstration videos have shown promise, but a significant embodiment gap remains when the resulting policies are deployed on real robots. A common remedy is to fine-tune these models on robot-specific demonstrations. However, robot data collection can be prohibitively expensive and time-consuming, which is particularly acute in dexterous manipulation, e.g., teleoperating a multi-fingered hand for even a single atomic task can take days. To address this, we introduce Dexterous Point Policy, a framework that learns dexterous manipulation policies directly from human videos and requires no robot demonstrations. Our core insight is that a unified 3D keypoint representation can bridge human and robot embodiments when used for both observations and actions. Specifically, we extract 3D keypoints of task-relevant objects and human hands from raw videos, and train an autoregressive transformer over these keypoints. We observe that at the keypoint level, specifically the wrist and fingertips, human and robot behaviors closely align, enabling direct policy transfer. On a suite of real-robot tasks spanning pick-and-place and tool use, Dexterous Point Policy attains 75.0% success, whereas a state-of-the-art VLA baseline reaches only 1.0%. Furthermore, our method generalizes strongly to unseen scenarios, including multi-object environments and novel object categories.
