Search papers, labs, and topics across Lattice.
This paper introduces World Tokens, an innovative architecture for embodied policies that integrates world modeling during training to enhance action prediction without incurring high inference costs. By employing a World Adapter that transforms visual-language model (VLM) features into a fixed set of world tokens, the method allows for effective conditioning of future-video denoising and action generation. The results demonstrate that World Tokens outperforms existing benchmarks, achieving competitive performance on LIBERO and superior real-world success rates while maintaining efficient deployment.
World Tokens achieves state-of-the-art performance in embodied action tasks without the heavy inference burden of traditional world models.
Vision-language-action (VLA) models are a widely adopted paradigm for embodied policies. They excel at efficient closed-loop control but do not explicitly model how physical scenes evolve as a task unfolds. Recently emerging world-action models (WAMs) leverage pretrained video world models to capture spatiotemporal evolution, yet retaining future generation or a large video backbone in the control loop substantially increases inference cost. We introduce World Tokens, an embodied policy architecture built around a World Adapter that bridges visual-language understanding, world-dynamics modeling, and action generation. It uses world modeling during training to enhance the action policy while preserving efficient deployment. Specifically, the World Adapter transforms VLM features into a fixed set of world tokens, which condition a jointly fine-tuned future-video denoiser and simultaneously serve as the action expert's sole visual-language context. This shared conditioning allows gradients from future-video denoising to directly shape the representation used for action prediction, while exclusive routing prevents the policy from bypassing that representation. At deployment, the world-model branch is removed, leaving only the VLM, World Adapter, and action expert, with no online video-model inference. With a 2B backbone and no embodied action pretraining, World Tokens is highly competitive on LIBERO, attains the best reported averages on SIMPLER, substantially improves real-world R1 Pro success over a matched action-only baseline, and generates each action chunk at VLA-level latency.
