Search papers, labs, and topics across Lattice.
The paper introduces "Stable Counting Capacity," a novel assay that probes the ability of LLMs to reliably count repeated symbols, isolating procedural reliability from knowledge dependencies and semantic confounds. Experiments across 100+ models reveal that their stable counting capacity falls significantly short of their advertised context limits, indicating a reliance on a limited set of internal "count-like" states. Once these states are exhausted, models resort to guessing, suggesting that fluent performance does not guarantee general rule-following.
LLMs can't reliably count beyond a small number of steps, revealing a surprising brittleness in their ability to execute seemingly simple procedures despite fluent performance on complex tasks.
Large language models perform strongly on benchmarks in mathematical reasoning, coding and document analysis, suggesting a broad ability to follow instructions. However, it remains unclear whether such success reflects general logical competence, repeated application of learned procedures, or pattern matching that mimics rule execution. We investigate this question by introducing Stable Counting Capacity, an assay in which models count repeated symbols until failure. The assay removes knowledge dependencies, semantics and ambiguity from evaluation, avoids lexical and tokenization confounds, and provides a direct measure of procedural reliability beyond standard knowledge-based benchmarks. Here we show, across more than 100 model variants, that stable counting capacity remains far below advertised context limits. Model behavior is consistent neither with open-ended logic nor with stable application of a learned rule, but instead with use of a finite set of count-like internal states, analogous to counting on fingers. Once this resource is exhausted, the appearance of rule following disappears and exact execution collapses into guessing, even with additional test-time compute. These findings show that fluent performance in current language models does not guarantee general, reliable rule following.
