Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Multi-agent Egocentric World Model (ME-World), a novel framework for predicting first-person observations in multi-agent environments by modeling synchronized ego-stream generation conditioned on fine-grained actions and shared environment memory. It addresses the limitations of prior work by enabling agents to interact through nuanced actions and ensuring consistency across agent views, actions, and environment state updates. ME-World demonstrates significant improvements in shared-world consistency, action control, identity preservation, and video generation quality compared to existing multi-agent world modeling approaches.
Fine-grained embodied interactions between multiple agents can be effectively modeled by synchronizing ego-stream generation with cross-view action consistency and shared environment memory.
Egocentric world models predict first-person observations conditioned on an agent's actions, but most focus on a single agent. Real embodied settings often involve multiple agents that act and interact within a shared environment. Existing multi-agent world models rely on coarse actions like locomotion, camera control, or discrete commands, leaving fine-grained embodied interactions underexplored. We formulate multi-agent egocentric world modeling as synchronized ego-stream generation for multiple agents interacting through fine-grained actions in a shared world. This requires cross-view action consistency, shared-environment consistency, and consistent propagation of interaction-induced state updates. We propose Multi-agent Egocentric World Model (ME-World), which jointly denoises multiple ego streams in a shared token sequence, conditions each stream on all agents'target-view poses, and grounds generation with shared environment memory. We train and evaluate on real and synthetic multi-agent data and introduce shared-world consistency metrics for environment, update, and identity consistency. Experiments show ME-World improves shared-world consistency, action control, identity preservation, and video quality over existing methods.
NVIDIA