Search papers, labs, and topics across Lattice.
This survey paper reviews the rapidly growing field of robot learning from human video data, focusing on techniques for transferring skills and the data foundations that support them. It presents a hierarchical taxonomy categorizing transfer pathways (task, observation, action) and analyzes their coupling with different data configurations and learning paradigms. The survey also investigates human video datasets and video generation schemes, highlighting trends, challenges, and future research directions.
Learning robot skills from human videos offers a scalable path to generalist robotic systems, but faces challenges in bridging the gap between human demonstrations and robot execution.
A critical bottleneck hindering further advancement in embodied AI and robotics is the challenge of scaling robot data. To address this, the field of learning robot manipulation skills from human video data has attracted rapidly growing attention in recent years, driven by the abundance of human activity videos and advances in computer vision. This line of research promises to enable robots to acquire skills passively from the vast and readily available resource of human demonstrations, substantially favoring scalable learning for generalist robotic systems. Therefore, we present this survey to provide a comprehensive and up-to-date review of human-video-based learning techniques in robotics, focusing on both human-robot skill transfer and data foundations. We first review the policy learning foundations in robotics, and then describe the fundamental interfaces to incorporate human videos. Subsequently, we introduce a hierarchical taxonomy of transferring human videos to robot skills, covering task-, observation-, and action-oriented pathways, along with a cross-family analysis of their couplings with different data configurations and learning paradigms. In addition, we investigate the data foundations including widely-used human video datasets and video generation schemes, and provide large-scale statistical trends in dataset development and utilization. Ultimately, we emphasize the challenges and limitations intrinsic to this field, and delineate potential avenues for future research. The paper list of our survey is available at https://github.com/IRMVLab/awesome-robot-learning-from-human-videos.
