Search papers, labs, and topics across Lattice.
This paper addresses the limitations of current video generators in maintaining a coherent world-state across segments by introducing Statebench, a benchmark that evaluates video continuation based on past-visible, occluded-process, and complex-transition states. The authors propose Stateagent, which uses an entity-state representation to update and ground the predicted state in response to new prompts, significantly enhancing the model's ability to generate coherent video continuations. Experimental results demonstrate a notable improvement in state tracking, with the all-case state score (SCS-All) increasing from 45.2 to 69.3, showcasing its effectiveness in both video continuation and story generation tasks.
Stateagent transforms video generation by effectively tracking and inferring world states, leading to a 54% improvement in coherence across segments.
Video generators build long videos by composing shorter parts, either by generating segments one after another or by autoregressively extending chunks. Each new part usually depends on memories of historical observations, such as recent frames, selected key frames, memory banks, or cached features. These memories preserve visible evidence from the past, but current generators do not reliably turn such evidence into a world-state interface: what holds in the video world after previous actions and how it should change under the next prompt. A past frame remains valid history, but it may not describe the state needed by the next segment; some states must instead be inferred from occluded or implicit changes rather than copied from a directly observed frame. This creates a simple but overlooked question for video continuation: given a previous video, its prompt, and a new prompt, can a model generate a continuation that reflects the state determined by both the historical video and the new prompt? To answer this question, we introduce Statebench, a benchmark that targets this gap by testing continuations over three state categories: past-visible states, occluded-process states, and complex-transition states. We further propose Stateagent, which explicitly maintains an entity-state representation, updates it under the new prompt, grounds the predicted post-action state as a future end frame, and renders the next video. Experiments show that our method improves controlled video continuation by raising the all-case state score (SCS-All) from 45.2 to 69.3, and also benefits story generation at the one-minute scale. Code is avaliable at https://github.com/AMAP-ML/StateAgent.
