Search papers, labs, and topics across Lattice.
The paper introduces the AGI Maze Prediction Datasets, a benchmark designed to evaluate the ability of Transformers to model world dynamics through various predictive tasks. By utilizing procedurally generated grid worlds, the benchmark distinguishes between learning transferable dynamics and mere memorization of familiar layouts, employing a unique training and evaluation methodology. Key findings reveal that a pseudo-video spatial-memory Transformer outperforms traditional models in fixed-horizon tasks, highlighting the importance of structured working memory in enhancing predictive capabilities.
A pseudo-video spatial-memory Transformer achieves perfect validation accuracy on complex tasks, revealing that structured working memory can outperform mere latent capacity in predictive modeling.
World modeling requires a predictive model to maintain and update an internal state adequate for reasoning about the consequences of actions. We introduce the AGI Maze Prediction Datasets and Benchmark, a lightweight controlled testbed for studying this capability in Transformers and other predictive models. Derived from procedurally generated, stateful grid worlds, the benchmark comprises per-step transition prediction, fixed-horizon state prediction, and sequential textual-observation prediction. Source-maze-disjoint training and validation splits, together with greedy exact-match evaluation, distinguish learning transferable action-conditioned dynamics from memorizing transitions in familiar layouts. We establish from-scratch byte-level Transformer baselines and compare them with two working-memory-augmented architectures. A generic auxiliary latent-memory Transformer can fit some training sets perfectly but does not consistently improve held-out performance. In contrast, a pseudo-video spatial-memory Transformer initializes a two-dimensional latent workspace from the input map and updates it from action history without receiving intermediate maps, positions, or state labels. Under the same data, objectives, and evaluation protocol, this model reaches perfect validation accuracy on selected fixed-horizon tasks where the byte and unstructured-memory baselines do not, and substantially improves sequential text-trace prediction. These results suggest that structured, task-aligned working memory can be more useful than additional latent capacity alone. More broadly, we argue that language grounding is mediated by persistent data structures and computations over them; the benchmark offers a compact setting for testing architectures that couple textual interfaces to learned structured state.
