Search papers, labs, and topics across Lattice.
This paper introduces Divergence-Adaptive Supervision Horizons (DASH), a novel approach that enhances on-policy self-distillation (OPSD) in reinforcement learning with verifiable rewards by adapting token-level supervision weights based on the temporal evolution of divergences during autoregressive generation. By mapping local distillation signals to adaptive propagation gates, DASH effectively captures the contextual nuances of discrepancies between teacher and student models, leading to improved reasoning capabilities. Experimental results demonstrate that DASH consistently outperforms standard OPSD across multiple mathematical reasoning benchmarks and model scales without requiring additional computational resources for teacher or student evaluations.
Adapting supervision weights based on the evolution of divergence histories boosts reasoning performance in language models without extra computational overhead.
Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models using automatically verifiable outcome signals, but these signals are typically sparse and at the sequence-level. On-policy self-distillation (OPSD) mitigates this sparsity by querying a privileged teacher at student-visited prefixes and providing dense token-level distributional supervision. Although this dense supervision alleviates signal sparsity, we find that standard OPSD still underexploits the temporal structure of the rollout. It assigns every local divergence the same coefficient, regardless of its position or the divergence sequence in which it occurs. In on-policy autoregressive generation, the same divergence magnitude can follow different discrepancy histories, reflecting different evolutions of the mismatch between the teacher and student. Since the local scalar alone cannot distinguish these temporal contexts, standard OPSD cannot adapt its token-level weights to the realized discrepancy sequence. To address this limitation, we propose Divergence-Adaptive Supervision Horizons (DASH). DASH maps the gap between each local distillation signal and the sequence-level mean to an adaptive propagation gate and then uses these gates to control backward multi-step aggregation. By doing so, DASH adjusts token-level supervision weights according to how local divergences evolve during generation. Experiments on three mathematical reasoning benchmarks across three model scales show that DASH improves over our matched vanilla OPSD reruns on every benchmark at all three scales. DASH reuses the teacher and student distributions that OPSD already computes, so the gains require no additional teacher or student forward pass. Code: https://github.com/DBtxy/DASH-OPSD
