Search papers, labs, and topics across Lattice.
This paper introduces LD4WAM, a novel framework that leverages motion-aligned latent dynamics to connect human video data with actionable low-level actions for World Action Models (WAMs). By employing a mixture-of-transformers architecture, LD4WAM achieves semantic reconstruction and real motion alignment, enabling effective action conditioning while maintaining the ability to generate future video frames. The model demonstrates strong performance in both simulation and real-world robotic tasks, showcasing its adaptability to diverse objects and environments.
Motion-aligned latent dynamics enable robots to learn actionable behaviors from human videos without losing visual fidelity across different embodiments.
Human video is playing an increasingly central role in training World Action Models (WAMs), owing to its diversity and low collection cost relative to teleoperated robot data. However, most WAMs learn from such video only by predicting pixel-level future frames, giving dynamics that are not directly actionable, whereas motion retargeting recovers directly actionable actions but leaves a large visual gap across embodiments. We therefore propose motion-aligned latent dynamics as an embodiment-agnostic representation to bridge video priors and low-level actions. We further present LD4WAM, which pairs a Latent Dynamics Model trained with semantic reconstruction and real motion alignment with a World Dynamics Action Model built as a mixture-of-transformers (MoT), which preserves full future-video generation and uses learnable queries to distill these latent dynamics from generated futures for action conditioning. Pretrained on our curated unified dataset of over 5{,}000 hours of human and robot data, LD4WAM performs strongly in RoboTwin simulation and on real robots equipped with both grippers and dexterous hands, while generalizing well to unseen objects and backgrounds.
