Search papers, labs, and topics across Lattice.
This paper introduces EasyOPD, a versatile on-policy distillation framework that enhances the distillation process for large language models by allowing for dynamic supervision on student-generated rollouts. By addressing the fragmentation in existing OPD methods, EasyOPD provides a unified backend that supports various supervision forms and tokenizer compatibilities, facilitating reproducibility and extensibility. Experimental results across diverse benchmarks indicate that EasyOPD maintains method-specific objectives while achieving competitive performance, thereby streamlining the distillation process in evolving model environments.
EasyOPD unifies on-policy distillation methods, enabling seamless integration and superior performance across diverse tasks in large language models.
Conventional language-model distillation often relies on fixed teacher-generated data, which may not cover the states encountered by an evolving student policy. On-policy distillation (OPD) instead collects teacher or evaluator supervision on student-generated rollouts. However, existing OPD methods differ substantially in supervision form, tokenizer compatibility, teacher access, and supervision granularity, leading to fragmented implementations that are difficult to reproduce and extend. We present \textsc{EasyOPD}, an on-policy distillation framework built on verl, a distributed reinforcement-learning framework for large language models. \textsc{EasyOPD} separates user-side configuration, method-specific supervision logic, and verl-based execution. Its method modules connect to the shared backend through extension boundaries for loss construction, rollout metadata, reward processing, tokenizer alignment, and teacher-side computation. We instantiate representative methods for three OPD settings -- cross-tokenizer OPD, on-policy self-distillation, and step-wise OPD. Experiments on reasoning, code-generation, scientific-knowledge, and tool-use benchmarks show that these implementations can be executed through the same verl-based backend while retaining their method-specific objectives and task-dependent performance profiles. We release \textsc{EasyOPD} with runnable YAML configurations, documentation, and an installable demonstration package and video.
NUS