Search papers, labs, and topics across Lattice.
The paper introduces Self-Referenced On-Policy Self-Distillation (SR-OPSD), which enhances on-policy self-distillation by addressing the instability caused by traditional moving target methods. By employing a token-level variational characterization and utilizing the R茅nyi divergence family, SR-OPSD effectively interpolates between a self-teacher policy and a reference policy, allowing for a more stable optimization process. Experimental results demonstrate that SR-OPSD achieves state-of-the-art performance in tasks such as scientific evaluation, mathematical reasoning, and code generation across multiple large language models.
SR-OPSD redefines the optimization landscape by stabilizing the distillation process, leading to superior performance in complex reasoning tasks.
On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on trajectories generated by the policy to be optimized, providing a useful complement to reinforcement learning with sparse outcome rewards. However, the self-teacher policy used in OPSD is typically a stop-gradient or exponential-moving-average copy of the policy conditioned on additional context information, and thus co-evolves with both the student policy and its on-policy context distribution. Directly matching such a moving target with a fixed projection objective can lead to unstable optimization or excessive distributional concentration. This nature of OPSD motivates the proposed \emph{Self-Referenced On-Policy Self-Distillation (SR-OPSD)}. At fixed student-generated contexts, a token-level variational characterization identifies the effective distillation target as a geometric interpolation between the self-teacher policy and a reference policy. Meanwhile, we use the R茅nyi divergence family to generalize the projection geometry. This formulation separates \emph{where} the adaptive target is placed from \emph{how} the student is projected toward it: the interpolation coefficient controls underlying target, while the R茅nyi order controls the projection geometry and its sensitivity to token-level density ratios. Extensive experiments across scientific evaluation, mathematical reasoning, and coding generation tasks with multiple large language models show that SR-OPSD achieves the state-of-the-art or competitive performance across various settings.
