Search papers, labs, and topics across Lattice.
The paper introduces on-policy prefix distillation, a modified on-policy distillation (OPD) method that applies the distillation objective only to prefixes of student-generated outputs, terminating sampling early. This approach leverages the observation that training signals in OPD are often concentrated in the output prefix. By focusing on prefixes, the method achieves comparable performance to full OPD while significantly reducing training FLOPs.
Get 2x-47x faster on-policy distillation by focusing only on the output prefix, without sacrificing performance.
On-policy distillation (OPD), which samples trajectories from the student model and supervises them with a teacher at the token level, avoids relying solely on verifiable terminal rewards and can yield better generalization than off-policy distillation. However, OPD requires expensive on-the-fly sampling of the student policy during training, which substantially increases training cost, especially for long responses. Our initial analysis shows that, during OPD, training signals are often concentrated in the prefix of each output, and that even a short teacher-generated prefix can significantly help the student produce the correct answer. Motivated by these observations, we propose a simple yet effective modification of OPD: we apply the distillation objective only to prefixes of student-generated outputs and terminate each sampling early during distillation. Experiments on a suite of AI-for-Math and out-of-domain benchmarks show that on-policy prefix distillation matches the performance of full OPD while reducing training FLOP by 2x-47x.
