Search papers, labs, and topics across Lattice.
This paper introduces the SHAPE framework, which analyzes Chain-of-Thought (CoT) trajectories in large language models (LLMs) through the lenses of semantic spaces and heuristics. The authors find that the mathematical heuristics employed by models are more predictive of answer correctness than traditional CoT features, and that models tend to concentrate their reasoning efforts within a few semantic spaces, mirroring human reasoning patterns. Additionally, the study shows that post-training LLMs to promote diverse heuristics significantly enhances their mathematical accuracy, providing a new diagnostic tool for understanding LLM reasoning.
Mathematical heuristics, not just reasoning paths, are the key to unlocking LLMs' accuracy in solving math problems.
Large language models (LLMs) achieve strong performance on mathematical reasoning benchmarks, yet the mathematically meaningful skills underlying their reasoning remain underexplored. We introduce SHAPE, a framework that analyzes Chain-of-Thought (CoT) trajectories through two lenses developed in mathematics education: (1) semantic spaces: the model's evolving mathematical interpretations of a problem (e.g., algebraic, geometric), and (2) heuristics: the specific mathematical actions taken within those spaces (e.g., simplifying the problem, working backward). We first use SHAPE to analyze the reasoning patterns of various models. Our findings reveal that the mathematical heuristics employed by a model better explain final answer correctness than traditional CoT features. Furthermore, models are likely to reach correct solutions by concentrating their reasoning effort within a few semantic spaces rather than exploring many disparate ones -- a pattern consistent with human behavior. Next, we utilize the SHAPE lens to evaluate whether post-training truly enhances mathematical proficiency. We find that reinforcement learning induces mode-seeking in heuristic usage. Lastly, we post-train LLMs by promoting diverse heuristics and demonstrate its effectiveness in improving accuracy. Overall, SHAPE provides a theoretically-grounded diagnostic framework for decoding LLM reasoning and offers a new path toward post-training LLMs for math reasoning. The code for our model is available at https://github.com/holi-lab/SHAPE-of-CoT
