Search papers, labs, and topics across Lattice.
This paper introduces Distilled Reinforcement Learning (Distilled RL), a novel approach that integrates teacher supervision into the reinforcement learning objective to enhance knowledge transfer in large language model (LLM) post-training. By addressing the limitations of existing methods鈥攃oarse-grained supervision in RL and the dilemma of unconditional imitation in on-policy distillation鈥擠istilled RL employs techniques like reverse importance sampling with clipping and sequence-level geometric normalization. Experimental results demonstrate that Distilled RL significantly outperforms traditional reinforcement learning and on-policy distillation methods in knowledge transfer effectiveness, achieving superior performance metrics in both within-family and cross-family distillation scenarios.
Distilled RL achieves a breakthrough in LLM post-training by effectively transferring previously inaccessible knowledge, outperforming traditional methods in both accuracy and adaptability.
Large language model (LLM) post-training is essential for improving reasoning, adaptation, and alignment. Existing methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD). However, RL relies on coarse-grained outcome supervision, resulting in difficult credit assignment and limited capability to acquire new knowledge. OPD, meanwhile, unconditionally matches teacher logits through KL divergence, which creates a dilemma: similar teachers provide little new knowledge, while substantially different teachers often yield ineffective guidance, largely restricting OPD to within-family distillation. We propose Distilled Reinforcement Learning (Distilled RL), which integrates teacher supervision into the RL objective to provide fine-grained guidance, selectively transfer new knowledge and avoid unconditional imitation. Distilled RL contains three components: reverse importance sampling with clipping, negative sample reset, and sequence-level geometric normalization. Through a concise and interpretable case study, we demonstrate that Distilled RL can effectively transfer previously unavailable knowledge from a teacher model to a student model. Extensive experiments across both within-family and cross-family distillation settings show that Distilled RL substantially outperforms standard RL and OPD in terms of both pass@1 and pass@k. Our code is available at https://github.com/597358816/Distilled-RL.
