Search papers, labs, and topics across Lattice.
This paper identifies a critical flaw in on-policy distillation (OPD): while improving task accuracy, it leads to severe overconfidence due to a mismatch between training and deployment contexts. They theoretically formalize this issue, demonstrating that teacher-conditioned success isn't a valid target for deployment-time confidence. To address this, they introduce Calibration-Aware OPD (CaOPD), which estimates empirical confidence from model rollouts and uses it as a distillation target, achieving Pareto-optimal calibration without sacrificing capability across various tasks and models.
On-policy distillation makes language models more accurate, but also dangerously overconfident, revealing a fundamental tension between capability and calibration.
On-policy distillation (OPD) is an increasingly important paradigm for post-training language models. However, we identify a pervasive Scaling Law of Miscalibration: while OPD effectively improves task accuracy, it systematically traps models in severe overconfidence. We trace this failure to an information mismatch: teacher supervision is formed under privileged context available during training, whereas the deployed model must report confidence using only deployment-time information. We formalize this perspective theoretically, showing that teacher-conditioned success is generally not a valid target for deployment-time confidence and that helpful privileged context induces entropy collapse and a systematic optimism bias. To address this, we propose a calibration-aware OPD framework, CaOPD, that estimates empirical confidence from model rollouts, replaces self-reported confidence with this student-grounded target, and distills the revised response through the same self-distillation pipeline. Experiments across various models and domains show that CaOPD achieves Pareto-optimal calibration while maintaining competitive capability, generalizing robustly under out-of-distribution and continual learning. Our findings highlight that capability distillation does not imply calibrated confidence, and that confidence should be treated as an essential objective in post-training. Code: https://github.com/SalesforceAIResearch/CaOPD
