Search papers, labs, and topics across Lattice.
This paper introduces a controlled benchmarking framework to evaluate the reasoning robustness of Large Language Models (LLMs) across nine classical reasoning tasks with explicit validity constraints and varying complexity. The study reveals a "reasoning collapse" phenomenon where LLM accuracy sharply declines beyond task-specific complexity thresholds, often exceeding 50%. The authors observed inconsistent reasoning traces, constraint violations, and confidently incorrect outputs, highlighting the limitations of LLMs in handling complex reasoning problems.
LLMs hit a "reasoning collapse" where accuracy plummets by over 50% on classical reasoning tasks once complexity exceeds a surprisingly low threshold, even with deterministic validation.
Large Language Models (LLMs) are increasingly described as possessing strong reasoning capabilities, supported by high performance on mathematical, logical, and planning benchmarks. However, most existing evaluations rely on aggregate accuracy over fixed datasets, obscuring how reasoning behavior evolves as task complexity increases. In this work, we introduce a controlled benchmarking framework to systematically evaluate the robustness of reasoning in Large Reasoning Models (LRMs) under progressively increasing problem complexity. We construct a suite of nine classical reasoning tasks: Boolean Satisfiability, Cryptarithmetic, Graph Coloring, River Crossing, Tower of Hanoi, Water Jug, Checker Jumping, Sudoku, and Rubik's Cube, each parameterized to precisely control complexity while preserving underlying semantics. Using deterministic validators, we evaluate multiple open and proprietary LRMs across low, intermediate, and high complexity regimes, ensuring that only fully valid solutions are accepted. Our results reveal a consistent phase transition like behavior: models achieve high accuracy at low complexity but degrade sharply beyond task specific complexity thresholds. We formalize this phenomenon as reasoning collapse. Across tasks, we observe substantial accuracy declines, often exceeding 50%, accompanied by inconsistent reasoning traces, constraint violations, loss of state tracking, and confidently incorrect outputs. Increased reasoning length does not reliably improve correctness, and gains in one problem family do not generalize to others. These findings highlight the need for evaluation methodologies that move beyond static benchmarks and explicitly measure reasoning robustness under controlled complexity.
