Search papers, labs, and topics across Lattice.
This paper critiques the effectiveness of on-policy self-distillation (OPSD) for long chain-of-thought (long-CoT) reasoning in large language models, revealing that traditional OPSD methods lead to rote memorization rather than meaningful reasoning improvements. The authors identify that the teacher's supervision signal is skewed towards reference-specific shortcuts, undermining the model's reflective reasoning capabilities. By introducing a two-step solution that isolates and transforms the relevant supervision components using pointwise mutual information, they achieve consistent performance gains across multiple long-CoT models while maintaining their natural epistemic behavior.
OPSD can backfire, leading to rote memorization instead of enhancing reasoning, but a novel decomposition approach reveals a path to meaningful improvements.
On-policy self-distillation (OPSD) has emerged as a promising paradigm for improving LLM reasoning, where a privileged teacher with access to reference solutions provides token-level supervision on the student's own generated trajectories. However, we find that OPSD consistently fails on long chain-of-thought (long-CoT) reasoning models, yielding at best marginal gains while destabilizing the reflective reasoning capability these models depend on. Through a novel decomposition of the teacher's supervision signal, we identify the root cause: the teacher's supervision is dominated by a reference-induced component that drives rote memorization of reference-specific shortcuts, while the question-conditioned, inference-transferable component is ignored or actively opposed. Based on this diagnosis, we propose a two-step solution. First, we construct a reference-only teacher (the same model conditioned on the reference without the question) to isolate the non-transferable component of the supervision signal; the residual after subtracting this component captures the question-conditioned, inference-transferable correction. Second, we use pointwise mutual information (PMI) as the mechanism to transform this residual into a well-formed PMI target distribution that the student can directly distill from, filtering out the reference-induced shortcut. Experiments on four long-CoT models across two datasets demonstrate consistent improvements over both the base model and standard OPSD, while preserving the models'natural epistemic behavior throughout training.
DAMO