Search papers, labs, and topics across Lattice.
This paper introduces an information-theoretic framework to analyze reasoning in LLMs, decomposing it into procedural information and epistemic verbalization (externalized uncertainty). It finds that epistemic verbalization, rather than specific trigger tokens, is crucial for sustained information acquisition and strong reasoning. The framework explains "Aha" moments and post-training effects, suggesting that LLMs benefit from explicitly managing and communicating their uncertainty during reasoning.
LLMs' "Aha!" moments aren't about magic tokens, but about explicitly verbalizing and managing uncertainty during reasoning, which drives performance.
LLMs often exhibit Aha moments during reasoning, such as apparent self-correction following tokens like "Wait," yet their underlying mechanisms remain unclear. We introduce an information-theoretic framework that decomposes reasoning into procedural information and epistemic verbalization - the explicit externalization of uncertainty that supports downstream control actions. We show that purely procedural reasoning can become informationally stagnant, whereas epistemic verbalization enables continued information acquisition and is critical for achieving information sufficiency. Empirical results demonstrate that strong reasoning performance is driven by uncertainty externalization rather than specific surface tokens. Our framework unifies prior findings on Aha moments and post-training experiments, and offers insights for future reasoning model design.
