Search papers, labs, and topics across Lattice.
This paper introduces WDL-OPD, a novel mixture-constrained co-training method for on-policy distillation that stabilizes the feedback loop between student and teacher policies. By employing an anchor policy for rollouts and an auxiliary policy for state evaluation, the method effectively aligns student outputs with a frozen teacher using reverse KL divergence. Experimental results demonstrate that WDL-OPD significantly improves performance on tasks such as MATH500 and code generation, outperforming traditional single-policy OPD configurations and supporting a stabilization hypothesis for policy training.
WDL-OPD boosts MATH500 accuracy from 0.630 to 0.685, showcasing a powerful new approach to stabilizing on-policy distillation.
On-policy distillation (OPD) aligns a student with a teacher on trajectories sampled from the student itself, reducing the train-test state mismatch of offline distillation. The same feedback loop can nevertheless be unstable: each update changes both the policy and the states on which the next update is computed. We introduce WDL-OPD, a mixture-constrained co-training method with two trainable policies. An anchor policy generates every rollout, an auxiliary policy evaluates the same visited states, and a geometric mixture of their token distributions is matched to a frozen teacher by reverse KL. Both policies receive gradient. We show that freezing the auxiliary recovers an anchor-plus-contrast proxy target closely related to OPD$^2$ and W2S-OPD, whereas joint training creates branch-level degrees of freedom that a static delta cannot express. In recorded Qwen3 experiments at 1.7B and 4B scale, WDL-OPD produces the strongest student checkpoint in each of four scale-domain settings. It raises MATH500 accuracy from 0.630 to 0.685 at 4B and from 0.521 to 0.585 at 1.7B. In code generation, seven single-policy OPD configurations exhibit entropy growth or trajectory degradation, while co-training reaches independently re-evaluated development scores of 0.637 and 0.375. Because several comparisons differ in curriculum or initialization, these results support a stabilization hypothesis rather than a universal causal claim. We provide the exact training algorithm, failure evidence, and the controlled comparison matrix needed to test that hypothesis.
