Search papers, labs, and topics across Lattice.
This paper introduces On-Policy Delta Distillation (OPD虏), a novel method in reinforcement learning that utilizes a delta signal to enhance on-policy distillation by capturing the differences between a teacher model and its base model prior to instruction tuning. By providing token-level supervision, OPD虏 effectively transfers reasoning capabilities from the teacher model, leading to significant improvements in performance across various benchmarks in mathematics, science, and code reasoning. The empirical results demonstrate that OPD虏 outperforms traditional on-policy distillation methods, enabling large language models to achieve high reasoning performance with minimal post-training time.
Capturing the delta between teacher and base models can revolutionize on-policy distillation, leading to superior reasoning capabilities in LLMs with just a brief post-training phase.
On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model. Although on-policy distillation has been studied and applied across various settings, its fundamental design remains underexplored. In this paper, we introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher's output distribution. The delta signal is defined as the difference between the teacher model and its base model prior to instruction tuning for reasoning capability. It therefore captures the changes induced by reasoning tuning and provides a more direct signal for transferring reasoning capabilities. Using extensive empirical evidence, we show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD$^2$). Experiments across mathematics, science, and code-reasoning benchmarks demonstrate that OPD$^2$ consistently outperforms conventional on-policy distillation, enabling reasoning LLMs to achieve strong performance with only a short post-training period. Code will be available at https://github.com/naver-ai/opd2
