Search papers, labs, and topics across Lattice.
This paper introduces Robust-WAM, a novel method that enhances World-Action Models (WAMs) by integrating a semantic foresight alignment objective into the action prediction stream while retaining the benefits of large-scale video generation model (VGM) pretraining. By employing learnable query tokens that align future-scene semantics with action outputs, Robust-WAM effectively addresses the fragility of action predictions under visual shifts, such as illumination changes. Experimental results demonstrate that Robust-WAM significantly improves out-of-distribution generalization success rates across various benchmarks and real-robot applications without compromising in-distribution performance.
Robust-WAM achieves superior out-of-distribution generalization in robot control by seamlessly integrating semantic foresight into action predictions while leveraging extensive VGM pretraining.
Mainstream World-Action Models (WAMs) adapt pretrained video generation models (VGMs) for robot control, transferring their learned dynamics prior for action prediction. These VGMs are typically trained in a variational autoencoder (VAE) latent space. However, the VAE latent space is optimized for pixel reconstruction, which rewards fine appearance detail and leaves the action prediction fragile under visual shifts. Recent works build WAMs in semantic latent space, which are more robust to appearance shifts. However, these models cannot leverage the large-scale VGM pretraining that exists only in VAE space. To overcome this dilemma, we propose Robust-WAM, a general post-training method for video-generation-based WAMs that preserves the VAE-based generative path and adds a lightweight semantic foresight alignment objective on the action stream. This retains the large-scale VGM pretraining while grounding actions in appearance-invariant dynamics that stay reliable under illumination shifts and other visual out-of-distribution conditions. Specifically, we employ learnable query tokens to bring future-scene semantics into the action stream by aligning their output hidden states with the semantic foresight of future ground-truth frames. To establish the temporal correspondence between each query and the future step it describes, we give it the positional encoding of the matching action tokens. Experiments on out-of-distribution generalization simulation benchmarks and a real-robot setup show that our Robust-WAM consistently improves the success rates of multiple WAM baselines without sacrificing in-distribution performance.
