Search papers, labs, and topics across Lattice.
The paper introduces ADWIN, an adaptive-window framework for on-policy distillation (OPD) that dynamically adjusts the rollout length used for training based on the alignment between student-induced prefixes and full teacher rollouts. ADWIN optimizes the accuracy-compute trade-off by focusing supervision on aligned prefixes, reducing the computational cost associated with full-rollout training. Experiments on math and code reasoning benchmarks demonstrate that ADWIN achieves comparable or better accuracy than full-rollout OPD while reducing training costs by up to 4.1x.
Stop wasting compute on full rollouts: ADWIN dynamically adapts on-policy distillation windows, slashing training costs by up to 4.1x without sacrificing accuracy on reasoning tasks.
On-policy distillation (OPD) transfers reasoning behavior by training a student on teacher feedback along student-generated trajectories, but standard full-rollout training ties every update to a costly completion and can over-allocate supervision to late positions with low marginal value for the current student. We revisit this assumption through the useful supervision horizon: student-induced rollouts can drift from teacher-preferred continuations, while aligned prefixes may already preserve the long-horizon OPD update direction. We propose ADWIN, an adaptive-window framework for OPD that treats rollout length as an online admissibility decision, training on short teacher-anchored prefixes while using delayed full-rollout probes to audit prefix--full alignment and adapt the next horizon with staleness control. Across math and code reasoning benchmarks in single-task, multi-task, and strong-to-weak settings, ADWIN improves the accuracy--compute trade-off over full-rollout OPD and prefix-based baselines, reducing end-to-end training cost by up to 4.1 times while achieving comparable or better accuracy.
