Search papers, labs, and topics across Lattice.
This paper introduces Relay On-Policy Distillation (Relay-OPD), which addresses the prefix failure issue in on-policy distillation by allowing a teacher model to briefly intervene during critical points in the student's trajectory. By leveraging a teacher-student continuation asymmetry, Relay-OPD effectively redirects the student's reasoning when missteps occur, resulting in improved trajectory optimization. The method demonstrates significant performance gains, achieving the best or second-best results across eight mathematical reasoning benchmarks, with an average improvement of 5.73% over standard OPD and a 50% reduction in training trajectory length.
Relay-OPD not only corrects reasoning missteps in real-time but also slashes training time by over half, setting a new standard for on-policy distillation efficiency.
On-policy distillation (OPD) grounds token-level supervision in the student's own trajectory, yet suffers from prefix failure: once the student commits to a wrong reasoning direction, all subsequent generation builds on this deviation, producing misdirected continuations that elicit unreliable supervision and waste compute. We identify a teacher-student continuation asymmetry on failed prefixes, where the teacher tends to redirect while the student continues along the original direction, and convert it into a label-free handoff trigger in Relay On-Policy Distillation (Relay-OPD). During training, Relay-OPD constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the resulting trajectory. A limited relay budget concentrates intervention on critical early positions while limiting departure from the student policy. With a Qwen3-4B-Instruct-2507 teacher and Qwen3-0.6B/1.7B-Non-Thinking students on eight mathematical reasoning benchmarks, Relay-OPD achieves the best or second-best results on every benchmark, outperforming standard OPD by +5.73% and the strongest baseline FastOPD by +1.49% on average for 1.7B, with consistent gains at 0.6B. Training trajectory length is reduced by over 50%.
