Search papers, labs, and topics across Lattice.
This study investigates the generalization behavior of on-policy distillation (OPD) in large language models by systematically varying generalization factors, including in-domain shifts and cross-domain transfer. The findings reveal that OPD primarily transfers reasoning behaviors from teachers rather than specific answers, with the relationship between teacher and student significantly influencing transfer efficacy. Notably, while same-origin teacher-student pairs enhance performance across various dimensions, cross-origin pairs tend to reinforce trained distributions, highlighting the complex interplay of teacher influences in multi-teacher settings.
OPD excels at transferring reasoning skills over specific answers, revealing a nuanced relationship between teacher-student origins that can complicate multi-teacher setups.
On-policy distillation (OPD) transfers teacher capabilities by supervising trajectories sampled from the student's own policy, yet its generalization behavior remains poorly understood, as most studies evaluate OPD on a single domain and on benchmarks close to the training data. We present a controlled study that varies one generalization factor at a time, from in-domain distribution shifts to cross-domain transfer and the multi-teacher setting. We find that OPD transfers a teacher's reasoning behavior rather than its answers to particular problems: training difficulty barely matters, and even problems the teacher never solves are useful. Transfer depends strongly on the origin relationship between teacher and student: same-origin pairs bring the student close to the teacher across languages, reasoning horizons, and even other domains, whereas cross-origin pairs mostly fit the trained distribution. This broad reach is a double-edged sword: since routing prompts to domain experts cannot confine each teacher's influence, combining them yields a mixture-dependent seesaw among their capabilities. These results clarify when OPD generalizes and offer a useful perspective for diagnosing multi-teacher OPD.
