Search papers, labs, and topics across Lattice.
This paper re-evaluates the GSM-Symbolic benchmark, which claimed to show reasoning flaws in LLMs via performance drops on template-generated math problems. Using more rigorous statistical testing (Generalised Linear Mixed Models), the authors show that the reported performance drops are not statistically significant for many models. They further identify a previously overlooked distributional shift in the GSM-Symbolic dataset related to the magnitude of numbers, which explains performance drops in many of the remaining cases.
Claims that LLMs lack genuine reasoning based on the GSM-Symbolic benchmark are likely overstated, as many reported performance drops disappear under more rigorous statistical scrutiny and when controlling for dataset distributional shifts.
The GSM-Symbolic benchmark (Mirzadeh et al., 2025) reported consistent performance drops across 25 Large Language Models (LLMs) when tested on template-generated variants of GSM8K problems, concluding that the models lack genuine reasoning capabilities. We argue that this conclusion rests on shaky statistical ground. Re-evaluating 20 open-weight models using Generalised Linear Mixed Models with per-question random effects, we find that only half exhibit statistically significant performance changes under the original prompt format. Moreover, we identify a previously unacknowledged factor: the main GSM-Symbolic dataset contains a systematically shifted distribution of larger integers in problem texts relative to GSM-Base (K-S statistic = 0.12, p < 0.001), contradicting the original authors' claims. Controlling for this large number effect accounts for significance in roughly half the remaining cases. Among models with statistically significant performance deltas, we identify distinct, model-specific failure profiles - including fragility of variable binding, arithmetic limitations, and dual-task interference - underscoring that blanket claims about LLM reasoning are both statistically premature and mechanistically misleading.
