Search papers, labs, and topics across Lattice.
This paper introduces JoyAI-RA 0.5, a Vision-Language-World-Action (VLWA) framework that enhances robot manipulation learning by integrating heterogeneous data sources, including human egocentric video, simulation, and real robot data. By employing dual action alignment鈥攂oth implicit and explicit鈥攖he framework effectively infers latent actions and grounds trajectories in a unified action space, facilitating efficient learning of physical dynamics. The results on the AgiBot benchmark demonstrate significant performance improvements with increasing volumes of human pretraining data, indicating that weakly labeled human experiences can serve as a robust training signal for scalable manipulation capabilities.
Abundant weakly labeled human video data can be transformed into a primary training signal for robot manipulation, significantly enhancing performance across tasks.
Robot data is scarce, so generalist policies need to learn from heterogeneous sources, including human egocentric video, simulation, and real robots, which differ in supervision and embodiment, with action labels missing or mutually incompatible. Human egocentric data scale best but sit farthest from robot data, and naive pooling causes negative transfer rather than knowledge sharing. We propose JoyAI-RA 0.5, a generalist Vision-Language-World-Action (VLWA) framework that couples physical world-dynamics priors with visual semantics and scales manipulation learning across such data via dual action alignment. Implicit action alignment infers latent actions from visual transitions, enabling action-free human, simulation, and robot data to guide a latent-action-conditioned world model in learning physical dynamics. Explicit alignment grounds reliable human and robot trajectories in a unified physical action space through a canonical action representation and camera-frame chunk-relative end-effector actions. An inner-outer-loop reinforcement stage then pairs efficient task adaptation with foundation-policy improvement. On a real-world AgiBot benchmark, JoyAI-RA performs strongly on both seen tasks and unseen variations. The task score improves consistently as the volume of human egocentric pretraining data increases and shows no sign of plateauing at our largest scale. This suggests that abundant but weakly labeled human experience can be converted into a transferable training signal, making human video not merely a weak auxiliary source but a primary axis along which manipulation capability can be scaled. Project page can be found at https://joyai-ra-05.github.io/.
