Search papers, labs, and topics across Lattice.
This paper introduces the World Action Planner, a robot planning system that integrates Vision-Language Models (VLMs) with a multi-task pose-image conditioned world model to enhance decision-making in diverse environments. By enabling agents to propose and iteratively refine action plans through optimization and search over imagined rollouts, the system demonstrates significant improvements in performance across various compositional tasks and zero-shot generalization scenarios. The results indicate that this approach outperforms existing state-of-the-art end-to-end policy models, providing a robust solution for generalizable decision-making in robotics.
Achieving superior zero-shot generalization, the World Action Planner outperforms leading policy models by leveraging the reasoning power of Vision-Language Models.
Building generalizable agents for diverse applications remains a fundamental challenge. While imitation learning-based policies succeed in specific training environments, they often fail to generalize to novel scenes and tasks. In this work, we propose World Action Planner, a robot planning system that leverages the reasoning capabilities of Vision-Language Models (VLMs) and the physical grounding of a multi-task pose-image conditioned world model. Our system enables an agent to propose initial action plans and iteratively refine them via optimization and search, reasoning over imagined world model rollouts. We demonstrate that our approach achieves superior performance across compositional tasks, new layouts, and zero-shot generalization scenarios, significantly outperforming state-of-the-art end-to-end policy models such as VLAs and WAMs. Project website at worldactionplanner.github.io
