Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a latent-action-based framework that leverages unlabeled egocentric human manipulation videos to train Vision-Language-Action (VLA) models without the need for extensive annotated datasets. By employing a Hybrid Disentangled VQ-VAE architecture, the authors successfully separate motion dynamics from environmental contexts, creating a cross-embodiment action codebook that enhances action intent learning. The proposed method demonstrates competitive performance against state-of-the-art models while significantly reducing the amount of required annotated data for downstream adaptation, achieving effective training with just 50 trajectories.
Unlabeled human videos can rival traditional datasets in training VLA models, achieving competitive results with minimal annotated data.
Training generalist Vision-Language-Action(VLA) models typically requires massive, diverse robotic datasets with high-fidelity action annotations. While egocentric human manipulation videos are abundant and capture significant environmental diversity, the absence of action labels makes them difficult to use in conventional training paradigms. To address this, we propose a latent-action-based framework designed to extract general action priors from unlabeled human videos. The architecture features a Hybrid Disentangled VQ-VAE that decouples motion dynamics from environmental backgrounds through physical masks, enabling the construction of a cross-embodiment action codebook. By pre-training on human videos with the codebook, the VLM backbone learns deep representations of action intent. For adaptation to specific embodiments, we introduce an intent-perception decoupling strategy where the VLM predicts the action intent while a separate frozen visual encoder provides state-specific features to the action expert, thereby reducing action hallucinations. Results in simulation and real-world environments show that our method, pre-trained exclusively on unlabeled human videos, performs competitively with state-of-the-art VLA models trained on massive annotated datasets, requiring only 50 trajectories for downstream adaptation.