Search papers, labs, and topics across Lattice.
The paper introduces AGI Maze, a benchmark framework designed to evaluate world-modeling capabilities in agents by presenting grid-based maze tasks that require persistent representations of state and memory. The authors demonstrate that existing large language models (LLMs) struggle to maintain internal representations necessary for navigating these environments, highlighting a significant gap in their reasoning abilities when faced with partially observable and stateful tasks. An initial evaluation shows that while a baseline agent utilizing message history can improve performance, it still falls short of human-level problem-solving in these mazes, underscoring the limitations of current LLMs in complex reasoning scenarios.
Existing LLMs fail to maintain internal representations in maze environments, revealing critical limitations in their reasoning capabilities.
Large language models (LLMs) are powerful pattern-completion systems, but their default operating mode - predicting the next token from a static context - does not reliably produce persistent, manipulable representations of an external world. Many tasks that look like "reasoning" in text become substantially harder once the environment is partially observable, stateful, and requires memory and structured hypotheses about hidden state. AGI Maze is a lightweight framework for building such environments without requiring high-dimensional sensory inputs. It provides a family of grid-based maze tasks with a clean API and multiple difficulty regimes. The goal is to create benchmarks where agents must learn and use world state representations, not just infer a local rule over readily provided observations. We provide an initial evaluation of several vanilla LLMs on simple mazes showing that they fail to represent mazes internally at LLM inference time. We also introduce a baseline agent, which is allowed to use its message history as a working memory to construct descriptions of observations at agentic runtime. Although this can improve performance, it is still insufficient for an LLM agent to reliably solve even small mazes within a step budget that is more than enough for humans.
