Search papers, labs, and topics across Lattice.
This study critically evaluates the effectiveness of on-policy self-distillation for continual post-training in foundation models, revealing that while it can enhance in-domain specialization under stable conditions, it leads to significant forgetting and potential model collapse in out-of-distribution scenarios. The authors contrast self-distillation policy optimization (SDPO) with more conservative on-policy reinforcement learning methods, demonstrating that the latter better preserves prior capabilities. Key findings indicate that denser self-distillation exacerbates parameter and response space drift, challenging the notion that on-policy data alone suffices for effective continual learning.
On-policy self-distillation can accelerate specialization but risks severe forgetting and model collapse when faced with new data distributions.
Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities. Recent work suggests that on-policy learning can mitigate forgetting, with on-policy self-distillation emerging as a particularly attractive approach. In this work, we revisit this optimistic view through self-distillation policy optimization (SDPO). Our experiments show that SDPO can accelerate in-domain specialization when teacher signals are stable and well aligned, but it struggles to generalize to out-of-distribution scenarios. In continual post-training, SDPO exhibits stronger forgetting and can even collapse, whereas on-policy reinforcement learning methods such as GRPO adapt more conservatively and better preserve prior capabilities. Further analyses reveal that denser self-distillation induces larger drift in both parameter space and response space, and can amplify high-frequency formatting artifacts through a self-reinforcing teacher--student loop. These findings suggest that on-policy data alone is insufficient for continual learning. Dense self-distillation can accelerate specialization when teacher targets are stable and token-level supervision is reliable, but it should not be treated as a default stabilizer for continual post-training. Our code is available at https://github.com/Moenupa/SDPO-CL.
