Search papers, labs, and topics across Lattice.
This paper investigates how supervised fine-tuning (SFT), reinforcement learning (RL), and on-policy distillation (OPD) affect confidence calibration in large language models during chain-of-thought (CoT) reasoning. By employing a three-stage calibration framework, the authors reveal that OPD excels in pre-reasoning confidence, SFT is optimal for early stopping signals, and RL provides reliable aggregation signals post-commitment. The introduction of PosConf, a position-aware confidence strategy, demonstrates significant improvements in answer aggregation and early stopping, highlighting the importance of context-specific confidence utilization in reasoning models.
Confidence in reasoning models can be dramatically improved by strategically leveraging position-aware signals, leading to better performance in challenging tasks.
Large language models have made strong reasoning gains through supervised fine-tuning, reinforcement learning, and on-policy distillation, yet these post-training methods are usually evaluated only by final-answer accuracy. We study how they reshape confidence during reasoning. We introduce a three-stage calibration framework that evaluates confidence before, during, and after chain-of-thought generation, corresponding to difficulty estimation, early termination, and answer aggregation. Through a controlled comparison on mathematical reasoning benchmarks, we find that OPD provides the most useful pre-reasoning confidence, SFT gives the strongest online signal for early stopping, and RL produces the most reliable trace-level signal for aggregation. We further show that confidence reliability is position-dependent: RL confidence becomes informative after a path-commitment phase, while OPD confidence is useful early but can become inversely calibrated later. Based on this observation, we propose PosConf, a position-aware confidence strategy that uses confidence only from reliable relative-position intervals. PosConf improves RL answer aggregation by 6.1 points over majority voting and consistently improves OPD early stopping under tight token budgets, with gains up to 4.3 points by avoiding its later inverse-calibration region, showing that \emph{confidence in reasoning models should be used both stage-wise and position-awarely}. Our code is available at https://github.com/EIT-NLP/Post-Training-Calibration.
