Search papers, labs, and topics across Lattice.
This paper investigates on-policy self-distillation using a single model as both teacher and student, revealing that while it enhances pass@1 accuracy through token-level feedback from a correct demonstration, it inadvertently reduces output diversity. The authors find that this reduction in diversity stems from compounding biases in the self-distillation process, leading to flattened pass@k curves and diminished performance in out-of-distribution scenarios. Through theoretical analysis and empirical evaluation on graph path-finding and question-answering tasks, they demonstrate that self-distillation can amplify biases, concentrating model outputs on dominant modes at the expense of functional and semantic variety.
Self-distillation may boost accuracy but comes at the hidden cost of significantly reduced output diversity, risking performance in diverse scenarios.
On-policy self-distillation achieves strong pass@1 accuracy by using a single model as both teacher and student, with the teacher conditioned on a correct demonstration to provide dense token-level feedback. We show that this could come at a hidden cost: rollout diversity decreases and pass@k curves flatten (i.e., generating more rollouts fails to improve accuracy). We trace this to compounding biases in the design of self-distillation with sampled demonstrations. The teacher scores each student rollout while conditioned on a sampled correct rollout, channeling its feedback through the model's own biases. We theoretically analyze the optimal self-distillation policy and show that it tilts the base distribution by a pointwise conditional mutual information score between the student's rollout and the correct rollout used as context. Unlike the ideal optimal on-policy reinforcement learning (RL), which preserves probability ratios among equally correct rollouts, self-distillation can amplify existing probability gaps, concentrating mass on already-dominant modes. On a controlled graph path-finding task and science question-answering benchmarks, self-distilled models match or exceed RL on average performance but exhibit substantially lower functional and semantic diversity, failing on out-of-distribution settings that require diverse strategies.
Mila