Search papers, labs, and topics across Lattice.
This paper introduces Video2Sim2Real, a comprehensive framework that enables robots to autonomously acquire dexterous manipulation skills from a single human video. By reconstructing a simulator-ready digital twin and optimizing robot configurations based on object-centric keyframes, the method effectively bridges the sim-to-real gap through improved perception and interaction dynamics. The framework outperforms existing techniques in simulated task success and real-world transfer, showcasing its potential for advancing robotic dexterity in everyday tasks.
Achieving robust sim-to-real transfer, this framework allows robots to learn dexterous skills directly from human videos, outperforming traditional methods.
Human manipulation videos are a convenient and intuitive source for robot learning. However, directly transferring human dexterity to robots remains challenging due to perception errors and embodiment gap. To address this, we introduce Video2Sim2Real, a full-stack framework for autonomous skill acquisition from a single human manipulation video. Our framework first uses off-the-shelf foundation models to reconstruct a simulator-ready digital twin and extract robot and object motion priors. Rather than treating the extracted robot motion as a reliable reference throughout execution, our key idea is to recover and leverage the most fundamental sources of supervision from the demonstrated skill: We identify object-centric keyframes to optimize the corresponding robot configurations using object information from the simulator, and use these configurations as anchors that refine the robot motion such that it ultimately has the desired impact on the environment. To bridge the remaining sim-to-real gap, we introduce a sim-to-real strategy that decouples robustness to noisy and incomplete perception from variations in hand-object interaction dynamics. Specifically, we learn to recalibrate robot configurations from noisy real-world point clouds via IL, and leverage residual RL to perform local finger-level adaptations to ensure for robust and effective interactions. Finally, a collision-aware motion planning module enables spatial generalization to novel object configurations. Across several everyday manipulation tasks, Video2Sim2Real improves simulated task success, safety, and trajectory coherence over numerous baselines, and achieves better sim-to-real transfer than existing techniques. These results demonstrate a promising path toward autonomous dexterous skill acquisition from human videos.
CMU ML