Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
We introduce CriticOPD, a novel on-policy distillation technique that addresses the failure of standard OPD when teachers recognize but follow student errors in rollouts. By incorporating a critique mechanism, CriticOPD guides the student to revise incorrect solutions, restoring teacher recoverability and enabling more effective distillation. This approach significantly improves student model performance on mathematical reasoning tasks, particularly for challenging competition problems.
Teachers can be coaxed into recovering from their own errors during distillation, unlocking substantial gains in student model performance.
On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error. The resulting output, including the critique as privileged information, is scored by the teacher while the student uses the revision without the critique for training. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves OPD, with the largest gains on competition problems.