Search papers, labs, and topics across Lattice.
This paper introduces CAIP (Contrastive Action-Image Pre-training), a novel vision encoder that leverages large-scale egocentric video to extract action signals from human hand poses, addressing the data scarcity in robotic datasets. By utilizing 32,041 hours of human video alongside only 88 hours of robotic manipulation data, CAIP learns a unified action-image representation through a contrastive objective, significantly enhancing the performance of downstream visuomotor control policies. The method achieves over 30% performance improvement on complex dexterous manipulation tasks compared to existing state-of-the-art vision encoders, demonstrating its effectiveness in real-world applications.
Extracting action signals from 32,041 hours of human video enables CAIP to outperform leading vision encoders in robotic manipulation tasks by over 30%.
Existing vision encoders for robotics face a fundamental bottleneck: robotic datasets lack the scale necessary for large-scale pre-training. Prior work circumvents this data scarcity by turning to internet-scale image and language data or egocentric human video. While these models show promise, neither paradigm learns from paired vision and action data, which downstream visuomotor control policies require. However, robot trajectories, the most direct source of this paired signal, are not available at pre-training scale, motivating us to extract action signals from abundant human video instead. To this end, we introduce CAIP (Contrastive Action-Image Pre-training), a vision encoder that treats human hand poses from large-scale egocentric video as a proxy for end-effector actions. By extracting 3D hand keypoints, a representation that aligns naturally with downstream robot action spaces, CAIP learns a unified action-image representation through a contrastive objective. Leveraging 32,041 hours of egocentric human video and only 88 hours of robotic manipulation data, CAIP outperforms state-of-the-art vision encoders including DINOv2, SigLIP, MVP, and R3M. Evaluated on a challenging real-world dexterous manipulation setup using Dexmate Vega and Sharpa Wave hands, CAIP yields performance gains of more than 30% on tasks involving folding, pouring, and fine-grained manipulation. Our results show that our method of contrastive action-centric pre-training yields a scalable path to achieving robust visual representations better suited for physical interaction.
BAIR
NVIDIA