Search papers, labs, and topics across Lattice.
This paper addresses the challenge of continual post-training for small language models, where existing Self-Distillation Fine-Tuning (SDFT) methods fail due to their reliance on strong in-context learning. The authors identify distribution shift as the root cause and introduce Synthetic Self-Distillation (SynthSelf), which outsources in-distribution rewriting to a slightly larger model. SynthSelf, combined with off-policy supervision and rejection sampling, enables effective continual learning for small models, outperforming existing methods across various acquisition and retention benchmarks.
Small models can now learn new skills without forgetting old ones, thanks to a novel distillation technique that leverages a larger model to guide the learning process.
Continual post-training faces a dual challenge of acquisition and retention: models must learn new capabilities while preserving prior knowledge. While Self-Distillation Fine-Tuning (SDFT) mitigates this using self-supervision with in-context demonstrations, it relies on strong in-context learning and instruction-following abilities. Consequently, SDFT fails on smaller models critical for low-latency, on-device use. To enable continual learning at smaller scales, we first identify the root cause of failure. We show that poor acquisition and forgetting correlate linearly with the post-trained model's distribution shift, revealing the necessity of in-distribution learning. We thus introduce Synthetic Self-Distillation (SynthSelf), imitating SDFT's in-distribution rewrite by outsourcing the task to a minimally larger, same-family aligner. Coupled with off-policy supervision and rejection sampling, this provides the high-quality supervision small models require without breaking their original distribution. Evaluations spanning 4 acquisition datasets (including a mixture) and 14 retention benchmarks show SynthSelf consistently outperforms 6 methods (including off- and on-policy distillation) across 3 small model scales. Extensive further analyses confirm SynthSelf paves the way toward effective small-model continual learning.
