Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
This study investigates the generalization behavior of on-policy distillation (OPD) in large language models by systematically varying generalization factors such as in-domain shifts and cross-domain transfer. The findings reveal that OPD primarily transfers reasoning behaviors from teachers rather than specific answers, indicating that the training difficulty is less significant than previously thought. Additionally, the relationship between teacher and student origins critically influences transfer effectiveness, with same-origin pairs fostering broader generalization across languages and reasoning tasks, while cross-origin pairs tend to restrict performance to trained distributions.
OPD transfers reasoning skills rather than answers, revealing a complex interplay between teacher-student origins that can either enhance or hinder model capabilities.
On-policy distillation (OPD) transfers teacher capabilities by supervising trajectories sampled from the student's own policy, yet its generalization behavior remains poorly understood, as most studies evaluate OPD on a single domain and on benchmarks close to the training data. We present a controlled study that varies one generalization factor at a time, from in-domain distribution shifts to cross-domain transfer and the multi-teacher setting. We find that OPD transfers a teacher's reasoning behavior rather than its answers to particular problems: training difficulty barely matters, and even problems the teacher never solves are useful. Transfer depends strongly on the origin relationship between teacher and student: same-origin pairs bring the student close to the teacher across languages, reasoning horizons, and even other domains, whereas cross-origin pairs mostly fit the trained distribution. This broad reach is a double-edged sword: since routing prompts to domain experts cannot confine each teacher's influence, combining them yields a mixture-dependent seesaw among their capabilities. These results clarify when OPD generalizes and offer a useful perspective for diagnosing multi-teacher OPD.