Search papers, labs, and topics across Lattice.
The paper identifies a "detection-extraction gap" in chain-of-thought reasoning, where models internally "know" the answer long before they explicitly output it. By comparing free-form continuation with forced extraction, the authors show that the answer is recoverable from early model states, but standard decoding fails to extract it effectively. They then introduce Black-box Adaptive Early Exit (BAEE), which leverages free continuations for both detection and extraction, achieving significant reductions in generation length (70-78%) and accuracy improvements (1-5pp).
LLMs waste up to 88% of their chain-of-thought tokens generating *after* they've already internally determined the answer.
Modern reasoning models continue generating long after the answer is already determined. Across five model configurations, two families, and three benchmarks, we find that \textbf{52--88\% of chain-of-thought tokens are produced after the answer is recoverable} from a partial prefix. This post-commitment generation reveals a structural phenomenon: the \textbf{detection--extraction gap}. Free continuations from early prefixes recover the correct answer even at 10\% of the trace, while forced extraction fails on 42\% of these cases. The answer is recoverable from the model state, yet prompt-conditioned decoding fails to extract it. We formalize this mismatch via a total-variation bound between free and forced continuation distributions, yielding quantitative estimates of suffix-induced shift. Exploiting this asymmetry, we propose Black-box Adaptive Early Exit (\BAEE{}), which uses free continuations for both detection and extraction, truncating \textbf{70--78\% of serial generation} while \textbf{improving accuracy by 1--5\,pp} across all models. For thinking-mode models, early exit prevents post-commitment overwriting, yielding gains of up to 5.8\,pp; a cost-optimized variant achieves 68--73\% reduction at a median of 9 API calls. Code is available at https://github.com/EdWangLoDaSc/know2say.
