Search papers, labs, and topics across Lattice.
This study investigates the distinct representations of solvability knowledge and verbalization in large language models (LLMs) to understand their mathematical reasoning capabilities. By disentangling these two components, the authors reveal that while knowledge about solvability is encoded separately, verbalization is more susceptible to manipulation and fabrication. The findings indicate that prompting with unsolvability cues effectively reduces fabrication by altering verbalization, and that activation steering can be used to enhance model abstention from incorrect answers.
Verbalization, not knowledge, drives fabrication in LLMs, revealing a critical leverage point for improving mathematical reasoning accuracy.
Although LLMs have made significant progress in mathematical reasoning, determining whether a mathematical problem is solvable remains a fundamental yet challenging capability. While recent studies have probed internal representations of model solvability beliefs, verbalization has primarily been studied behaviorally rather than as an internal representation, limiting its analysis and manipulation. We address this gap by separately probing representations of solvability knowledge and verbalization, allowing us to disentangle the two within model hidden states. Across multiple LLMs, we show that knowledge and verbalization are encoded as distinct, linearly decodable representations and that fabrication is primarily associated with changes in verbalization rather than the underlying knowledge. Prompting with unsolvability cues reduces fabrication primarily by shifting verbalization, while activation steering demonstrates that these representations can be echanistically manipulated to improve model abstention.
