Search papers, labs, and topics across Lattice.
The paper introduces H$^2$SD, a hybrid hindsight self-distillation framework that enhances reinforcement learning with verifiable rewards (RLVR) by addressing the limitations of existing methods in token-level credit assignment. By leveraging a teacher model differently based on trajectory correctness, H$^2$SD modulates update magnitudes for successful trajectories and minimizes reverse KL divergence for failed ones, effectively balancing supervision and stability. Experimental results demonstrate that H$^2$SD outperforms traditional RLVR, OPSD, and RLSD approaches across various reasoning benchmarks, highlighting its efficiency and robustness in training large language models.
H$^2$SD achieves superior reasoning performance by intelligently adapting teacher signals based on trajectory outcomes, leading to more effective learning in large language models.
Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models on tasks such as mathematical reasoning and code generation. However, most RLVR methods assign a scalar outcome reward to an entire trajectory, resulting in sparse supervision and limited token-level credit assignment. On-policy distillation (OPD) provides denser supervision by distilling token-level distributions from a stronger teacher model, but requires an additional teacher and typically assumes a shared vocabulary. On-policy self-distillation (OPSD) removes this dependency by conditioning the same model on privileged information to construct a teacher policy. However, directly matching the teacher distribution may cause information leakage and unstable optimization. RLSD avoids direct matching by using the teacher signal only to modulate update magnitudes, but it cannot provide an explicit correction direction when the sampled reasoning fails. To address this tradeoff, we introduce $\mathrm{H}^{2}\mathrm{SD}$, a hybrid hindsight self distillation framework that uses the teacher differently according to trajectory correctness. For successful trajectories, the teacher receives the student response confirmed as correct together with a rephrasing instruction, and its probabilities on the original response tokens are used to modulate update magnitudes without changing the direction determined by the reward. For failed trajectories, we condition the teacher on a reference hint containing key reasoning steps and a verified answer, and minimize the reverse KL divergence from the student to the teacher. Experiments on multiple challenging reasoning benchmarks show that H$^2$SD consistently outperforms representative RLVR, OPSD, and RLSD baselines while maintaining stable optimization and favorable generation efficiency.
