Search papers, labs, and topics across Lattice.
This paper addresses the limitations of on-policy distillation (OPD) in scenarios where teacher models and supervised fine-tuning (SFT) data sources are inconsistent. By introducing Lightning OPD 2.0, which incorporates cross-fitted style residualization to mitigate style bias, the authors demonstrate that this method can effectively leverage teacher evidence despite discrepancies in wording and reasoning cadence. The results show that Lightning OPD 2.0 significantly outperforms its predecessor in cross-teacher settings, achieving notable performance improvements on mathematical reasoning and code generation benchmarks.
Lightning OPD 2.0 reveals that you can achieve superior performance in cross-teacher distillation by effectively mitigating style bias, even when teacher consistency is compromised.
On-policy distillation (OPD) provides dense token-level supervision from a teacher, but its effectiveness can depend on teacher consistency, meaning that the model providing OPD supervision should also have generated the demonstrations used to train the supervised fine-tuning (SFT) reference. However, this condition is frequently violated in practice when SFT data have mixed or unknown provenance or when different models are preferred for SFT data generation and subsequent distillation. In such cross-teacher settings, even a stronger OPD teacher can yield little improvement over the SFT reference. We find that raw teacher--reference disagreement contains potentially useful context-specific teacher evidence as well as a recurring component associated with differences in wording, formatting, and reasoning cadence. We introduce Lightning OPD 2.0 with cross-fitted style residualization, which uses rollout-level cross-fitting to estimate this recurring component as an operational proxy for style-token bias and subtracts it before constructing the token-level OPD update. Across mathematical reasoning and code generation benchmarks, Lightning OPD 2.0 consistently outperforms Lightning OPD in cross-teacher settings. Starting from Klear-Reasoner-8B-SFT, Lightning OPD 2.0 reaches 82.4% on AIME 2024 and 63.0% on LiveCodeBench v5. Together, these results establish Lightning OPD 2.0 as a practical approach to cross-teacher OPD, relaxing teacher consistency as a prerequisite and allowing the SFT data generator and distillation teacher to be selected independently. Code will be released soon.
