Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the ability of video world models to maintain unobserved world states using an action-conditioned video Shell Game, revealing that traditional models like Transformers and Mamba struggle with extrapolation beyond a training horizon of 5 swaps. Despite their capacity to render visually plausible video, these models fail to carry the hidden state necessary for accurate tracking, as the architectural limitations restrict state retention to an append-only KV cache. The authors identify two successful mechanisms for state extrapolation: linear attention with negative transition eigenvalues and a nonlinear fast weight mechanism that updates the feature map, providing insights into building more effective stateful video world models.
Traditional video world models falter in tracking hidden states beyond a limited training horizon, but two innovative mechanisms show promise for effective state retention and extrapolation.
Video world models are increasingly used as simulators, yet visual fidelity alone does not show that a model maintains the hidden state of the world. We examine this gap with an action-conditioned video Shell Game, a visual analog of $S_5$ state tracking that decouples visual rendering from compositing the hidden state underneath. Bidirectional and autoregressive Transformers, Mamba, and linear attention restricted to nonnegative transition eigenvalues all fit the training horizon of 5 swaps and then fall toward chance on longer swap chains (extrapolation) while still rendering plausible video with additional denoising steps providing no benefit. The pixel-based diffusion target never supervises the unseen hidden state, so the generated frames cannot carry it and the state has to live inside the architecture rather than in the tokens. For a Transformer, that architectural state is only an append-only KV cache, so the model has to re-derive the hidden arrangement from the whole history at every chunk. We find two mechanisms that do extrapolate, and both carry a state across chunks and revise it in place. Linear attention succeeds once its transition eigenvalues may be negative, and TTT with a nonlinear fast weight succeeds by updating the feature map through which it reads its own state. We further examine harder cases in dynamic world exploration tasks, and discuss the broader implications for building stateful video world models.