Search papers, labs, and topics across Lattice.
The paper introduces MultiWorld, a novel action-conditioned video generation framework for multi-agent environments that maintains multi-view consistency. It uses a Multi-Agent Condition Module for precise agent control and a Global State Encoder to ensure coherent observations across views. Experiments across multi-player games and multi-robot manipulation show MultiWorld achieves superior video fidelity, action-following, and multi-view consistency compared to existing methods.
Accurately simulating multi-agent interactions with consistent multi-view video is now possible thanks to MultiWorld, a framework that scales to many agents and viewpoints.
Video world models have achieved remarkable success in simulating environmental dynamics in response to actions by users or agents. They are modeled as action-conditioned video generation models that take historical frames and current actions as input to predict future frames. Yet, most existing approaches are limited to single-agent scenarios and fail to capture the complex interactions inherent in real-world multi-agent systems. We present MultiWorld, a unified framework for multi-agent multi-view world modeling that enables accurate control of multiple agents while maintaining multi-view consistency. We introduce the Multi-Agent Condition Module to achieve precise multi-agent controllability, and the Global State Encoder to ensure coherent observations across different views. MultiWorld supports flexible scaling of agent and view counts, and synthesizes different views in parallel for high efficiency. Experiments on multi-player game environments and multi-robot manipulation tasks demonstrate that MultiWorld outperforms baselines in video fidelity, action-following ability, and multi-view consistency. Project page: https://multi-world.github.io/
