Search papers, labs, and topics across Lattice.
This paper addresses the fragility of on-policy distillation (OPD) in long-horizon reasoning tasks by introducing blockwise policy-drift gating, a method that enhances student policy training through a lightweight drift controller. By computing log-probability shifts between the behavior student and the current student over fixed blocks, the method effectively reweights OPD position losses without altering teacher targets or rollout policies. The implementation of this approach yields a significant improvement in the mean pass@8 metric across multiple math reasoning benchmarks, demonstrating its effectiveness in enhancing solve-rate robustness.
Blockwise policy-drift gating boosts on-policy distillation performance, increasing solve rates by effectively managing policy drift without changing teacher signals.
On-policy distillation (OPD) trains a student policy using teacher signals computed on trajectories sampled by the student itself. Recent work shows that sampled-token OPD can be fragile on long-horizon reasoning tasks and that local teacher-support matching is a simple and effective repair. This paper introduces blockwise policy-drift gating, a lightweight student-only old-current drift controller for OPD under rollout reuse. The method computes log-probability shifts between the behavior student and the current student on the sampled token path, aggregates these shifts over fixed blocks or spans, and uses the resulting detached, mean-normalized gates to reweight OPD position losses. It does not change teacher targets, teacher top-K supports, or the rollout policy. In a six-variant Qwen3 math reasoning benchmark with a uniform 200-step training budget for all trained variants, we use pass@8 as the primary problem-level solve-rate metric. Fixed 64-token block gating improves sampled-token OPD mean pass@8 from 0.4978 to 0.5160 across AIME24, AIME25, MATH500, and AMC23. On Teacher-TopK/LSM, Block64 gives the best four-benchmark mean pass@8 among trained students. The results identify local old-current policy drift as a practical control signal for reused OPD rollouts and motivate block-level gating as a simple default for improving solve-rate robustness.
