Search papers, labs, and topics across Lattice.
To address the severe variance and late-stage training instability pervasive in Large Language Model on-policy distillation, the authors systematically analyze advantage dynamics and introduce a Total Variation-regulated distillation framework (TV-OPD). They uncover that student policy convergence depends predominantly on the sign鈥攔ather than the precise magnitude鈥攐f token-level advantages, meaning that bounded, smoothed signals can eliminate gradient noise without degrading target performance. Across multiple distillation benchmarks, TV-OPD consistently stabilizes optimization dynamics, eliminates late-stage degradation, and outperforms conventional on-policy baselines with significantly lower variance.
On-policy LLM distillation does not actually need precise advantage magnitudes: retaining merely the directional sign of token advantages matches standard distillation performance while Total Variation smoothing eliminates late-stage training instability.
On-Policy Distillation (OPD) facilitates the transfer of knowledge from domain expert to student in the post-training phase of Large Language Models (LLMs). However, the supervision signals in mainstream OPD methods suffer from high variance and noise which is generally instable during training. In this work, we systematically investigated what really matters to the performance and the fundamental mechanisms behind the instability during training. We found that retaining only the sign of token-level advantages is sufficient to achieve the performance comparable to standard OPD. Meanwhile, smoother and bounded advantages can stabilize the training process without sacrificing its performance. These motivated us to shape the advantages using the Total Variation (TV) and propose a robust TV regulated On-Policy Distillation (TV-OPD) method. Benefiting from the bounded and diminished advantages, TV-OPD exhibits stable training dynamics and steady late-stage performance. We conducted comprehensive experiments and found that, across various settings, TV-OPD consistently achieved better performance and lower variance in the late-stage of training.
