Search papers, labs, and topics across Lattice.
This survey provides a comprehensive overview of On-Policy Distillation (OPD) for LLMs, which addresses exposure bias in off-policy distillation by allowing students to generate their own trajectories and receive teacher feedback. The authors introduce a unified $f$-divergence framework to organize the OPD landscape along three key dimensions: feedback signal, teacher access, and loss granularity. The survey analyzes representative methods, industrial deployments, and open problems, providing a structured understanding of OPD's potential and challenges.
On-Policy Distillation could be the key to more robust and reliable LLM knowledge transfer, but the field is fragmented and lacks a unified theoretical understanding.
Knowledge distillation has become a primary mechanism for transferring reasoning and domain expertise from frontier Large Language Models (LLMs) to smaller, deployable students. However, the dominant paradigm remains \textit{off-policy}: students train on static teacher-generated data and never encounter their own errors during learning. This train--test mismatch, an instance of \textit{exposure bias}, causes prediction errors to compound autoregressively at inference time. On-Policy Distillation (OPD) addresses this by letting the student generate its own trajectories and receive teacher feedback on these self-generated outputs, grounding distillation in the theory of interactive imitation learning. Despite rapid growth spanning divergence minimization, reward-guided learning, and self-play, the OPD literature remains fragmented with no unified treatment. This survey provides the first comprehensive overview of OPD for LLMs. We introduce a unified $f$-divergence framework over on-policy samples and organize the landscape along three orthogonal dimensions: \emph{feedback signal} (logit-based, outcome-based, or self-play), \emph{teacher access} (white-box, black-box, or teacher-free), and \emph{loss granularity} (token-level, sequence-level, or hybrid). We systematically analyze representative methods, examine industrial deployments, and identify open problems including distillation scaling laws, uncertainty-aware feedback, and agent-level distillation.
