Search papers, labs, and topics across Lattice.
This paper addresses the challenge of diversity distillation failure in sampled-token on-policy distillation (OPD) by introducing Influence-Directed Adaptive On-Policy Distillation (IDA-OPD), which preserves entropy-expanding updates while adapting entropy-contracting ones. By leveraging First-Order Local Entropy Influence, the authors provide a novel framework that links entropy contraction to negative-influence positions, allowing for more effective capability transfer from teacher to student models. Experimental results demonstrate that IDA-OPD significantly enhances pass@$k$ performance and maintains pass@1, achieving these improvements at a lower computational cost compared to traditional methods.
IDA-OPD not only boosts diversity in distilled models but does so while cutting computational costs, challenging the notion that more data always leads to better performance.
Sampled-token on-policy distillation (OPD) efficiently transfers capabilities from teacher to student using student-generated tokens, requiring teacher probabilities only for sampled tokens. Yet it frequently suffers from diversity distillation failure: the student's pass@1 improves while its pass@$k$ plateaus, failing to inherit the teacher's diversity. To explain this, we introduce First-Order Local Entropy Influence, a signed first-order proxy that decouples each update's entropy effect into the teacher--student log-probability gap and the student's local probability structure, and empirically links entropy contraction to negative-influence positions. Motivated by this, we propose Influence-Directed Adaptive On-Policy Distillation (IDA-OPD): rather than relying on costly full-vocabulary Forward-KL objectives, it preserves entropy-expanding updates while replacing entropy-contracting ones with divergence-adaptive advantage shrinkage, using only the teacher's sampled-token log-probability. Experiments on reasoning-oriented distillation show IDA-OPD consistently improves pass@$k$, inheriting the teacher's diversity through distillation, matches the strongest teacher-informed methods at strictly lower cost, and broadly maintains vanilla OPD's pass@1, all without full-vocabulary teacher information.
