Search papers, labs, and topics across Lattice.
The authors introduce Chess-World-Model, a benchmark comprising 10 million chess games, to evaluate models' ability to track the exact board state from move sequences. They include both a real-game split and an out-of-distribution split generated from random legal moves to test generalization. Benchmarking causal Transformers, block-diagonal SLiCE, Mamba-3, and Gated DeltaNet, they find that recurrent models outperform Transformers, and that the random-uniform split exposes limitations even at 40 million parameters, highlighting the importance of expressive state-transition mechanisms.
Even at 40 million parameters, models struggle to accurately track chess states from random legal moves, revealing that scale alone can't overcome limitations in state-transition mechanisms.
World models require state tracking, which is the ability to maintain a correct latent state across action sequences. Existing benchmarks are often synthetic or language-based, limiting their value as tests of structured state updates in realistic domains. We introduce Chess-World-Model, a large-scale state-tracking benchmark built from 10 million real chess games, where models predict the exact board state reached after a sequence of legal moves. Alongside a held-out real-game split, we include an out-of-distribution split from uniformly random legal play, which tests whether models learn the transition rules rather than shortcuts from common human positions. Prior theoretical and empirical work has shown that Transformers struggle to state-track, while input-dependent linear RNNs require expressive state-transition matrices to do so. We therefore benchmark a causal Transformer, block-diagonal SLiCE, Mamba-3, and Gated DeltaNet with negative eigenvalues under a matched interface and training protocol. The recurrent models strongly outperform the Transformer at 3 and 8 million parameters. Real-game performance saturates above 18 million parameters, but the random-uniform split remains discriminative up to 40 million, exposing failures otherwise hidden by scale. Additionally, ablations show that less expressive state-transition mechanisms reduce performance on the out-of-distribution split for all three recurrent models. Together, these results establish Chess-World-Model as a practical large-scale benchmark for state tracking that exposes failures model scale would otherwise conceal.
