Search papers, labs, and topics across Lattice.
This paper introduces Vid2WAM, an offline distillation framework that enhances World Action Models (WAMs) by transferring visual diffusion priors from a large video foundation model to a compact WAM student. By employing task-conditioned future rollouts and an inverse dynamics model, Vid2WAM effectively supervises future predictions and action learning, while mitigating the impact of noisy pseudo-actions through source-aware residual action adaptation. Experimental results show that Vid2WAM significantly improves generalization to novel tasks and data efficiency, even when limited expert demonstrations are available, all while maintaining low-latency inference for practical deployment.
Vid2WAM achieves superior task generalization and data efficiency by leveraging video diffusion priors, even with minimal expert demonstrations.
World Action Models (WAMs) improve robot policy learning by jointly modeling future visual dynamics and actions. However, their scalability and generalization remain constrained by their reliance on costly expert demonstrations. We challenge this by asking whether future supervision for WAMs must originate from target-task expert trajectories. In this paper, we propose Vid2WAM, an offline distillation framework that transfers visual diffusion priors from a large video foundation model into a compact WAM student. Given an observation and language instruction, Vid2WAM distills supervision through two complementary channels: task-conditioned future rollouts directly supervise the student's future prediction branch, while an inverse dynamics model recovers embodiment-specific pseudo-actions for action learning. To robustly integrate synthetic and real supervision, we introduce source-aware residual action adaptation that learns source-specific corrections around a shared action backbone and mitigates interference from noisy pseudo-actions. During inference, both the video teacher and inverse dynamics model are discarded, leaving only the WAM student for efficient deployment. Simulation and real-world experiments demonstrate that Vid2WAM improves novel-task generalization and data efficiency under limited expert demonstrations while preserving low-latency inference.
