Search papers, labs, and topics across Lattice.
This paper introduces the Self-Distilled Policy Gradient (SDPG) framework, which leverages on-policy self-distillation to enhance reinforcement learning by using a language model to supervise its own outputs with privileged context. By employing a full-vocabulary student-to-teacher reverse Kullback-Leibler divergence loss, SDPG integrates group-relative verifier advantages and KL regularization to stabilize training and improve performance. Empirical results demonstrate that SDPG outperforms existing methods like RLVR and other self-distillation baselines, indicating its effectiveness in sparse-reward scenarios.
Self-distillation can significantly enhance reinforcement learning stability and performance, outperforming traditional methods in sparse-reward environments.
On-policy self-distillation, where a language model conditions on privileged context to supervise its own generations, is a promising source of dense supervision for sparse-reward reinforcement learning. Actually, it can be instantiated as an auxiliary full-vocabulary student-to-teacher reverse Kullback-Leibler divergence loss. We therefore propose SDPG, a self-distilled policy-gradient framework that combines group-relative verifier advantages with normalized standard deviation, exact full-vocabulary on-policy self-distillation, as well as reference-policy KL regularization. Empirically, SDPG improves stability and performance over RLVR and self-distillation baselines. The code is available at https://github.com/lauyikfung/SDPG.
