Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
Waypoint-OPD addresses limitations in policy-shift distillation by leveraging intermediate teacher checkpoints to assess the consistency of policy evolution during RL training. This method gates the transfer of endpoint policy shifts based on directional agreement between early and late action preference changes at student-visited prefixes, mitigating the impact of conflicting shifts. Experiments with Qwen3 models demonstrate that Waypoint-OPD significantly improves performance across mathematical and code reasoning tasks compared to endpoint-only distillation.
Gating policy distillation with directional agreement between early and late RL shifts unlocks more effective knowledge transfer, boosting LLM performance on reasoning tasks.
Policy-shift distillation reuses the difference between a teacher's log probabilities before and after reinforcement learning (RL) as supervision on student-generated trajectories. However, this endpoint policy shift cannot reveal whether local action preferences evolved consistently during teacher training. The same endpoint shift can result from early and late shifts that reinforce or partially cancel each other. We investigate whether agreement between early and late teacher policy shifts can guide selective transfer. We introduce Waypoint-guided On-Policy Distillation (Waypoint-OPD), which uses an intermediate teacher checkpoint from the same RL run to compare early and late shifts in relative action preferences at student-visited prefixes. We gate the endpoint shift according to the directional agreement between the early and late shifts, suppressing it when they point in opposing directions. Compared with endpoint-only transfer, our method requires only one additional frozen-teacher scoring pass. Using RL-trained Qwen3-4B teachers, we evaluate our method with Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B as student models in mathematical and code reasoning. In all six domain--student settings, Waypoint-OPD achieves the highest domain-average score among the compared distillation methods. Ablations with the Qwen3-8B student show that removing or shuffling gates reduces mathematical reasoning performance. These results support the use of directional agreement between early and late teacher policy shifts to weight the endpoint shift during student training.