Search papers, labs, and topics across Lattice.
This paper introduces DynaVieW, a dynamic schema-guided world model designed to enhance the temporal modeling of visual scenes in videos and multi-image sequences. By employing a mixture-of-experts architecture that integrates state-transition sequences and hierarchical schemas, DynaVieW effectively predicts and simulates visual dynamics, leading to significant improvements in tasks such as visual narrative creation and world simulation. The model demonstrates enhanced consistency, controllability, and instruction-following capabilities, addressing a critical gap in multimodal LLM performance in dynamic visual contexts.
DynaVieW reveals that a schema-guided approach can drastically improve the modeling of visual dynamics, leading to superior performance in narrative generation and simulation tasks.
Multimodal LLMs struggle to systematically model the temporal evolution of visual scenes in videos or multi-image sequences. Such inputs require models to predict or simulate multiple levels of dynamic constituents, such as actions taken in the visual sequence, and the associated changes to the visual environment that result. To address this challenge, we propose a dynamic schema-guided world model, DynaVieW, optimized for visual dynamic prediction and simulation. DynaVieW achieves an in-depth understanding of visual dynamics by learning interleaved state-transition sequences, where states cover broad visual scenes from video keyframes, and transitions capture comprehensive dynamic constituents within a hierarchical schema. DynaVieW jointly models transition prediction and state simulation under a mixture-of-experts architecture, with a cross-expert selective attention and a schema token re-weighted loss, to ensure effective and robust learning. DynaVieW's understanding of visual dynamics boosts its downstream performance in visual narrative creation and world simulation, showing improved consistency, controllability, and instruction-following.
