Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the effects of on-policy distillation (OPD) on model parameters, revealing that OPD updates are both small and coordinate-sparse, predominantly affecting feedforward networks across various language and vision-language models. The authors find that training only the identified subnetwork can achieve performance nearly equivalent to full OPD, while noting that the sparsity-inducing SGD optimizer performs worse than AdamW due to the latter's ability to handle heterogeneous gradient scales. Additionally, the updates are shown to be numerically full-rank but concentrated away from principal singular subspaces, indicating that dense teacher supervision preserves distinct geometric characteristics of on-policy post-training.
Sparse updates in on-policy distillation can match full performance with significantly reduced training overhead, challenging conventional wisdom about dense parameter updates.
On-policy distillation (\textsc{OPD}) has recently become a prominent post-training recipe as it combines two desirable ingredients: on-policy student trajectories and dense teacher supervision, yet how this hybrid changes a model's parameters remains unclear. Across several language and vision-language model pairs and use cases, our analysis yields two main findings. On sparsity, \textsc{OPD}-style updates are small and coordinate-sparse. They are distributed across layers and are usually FFN-heavy. This sparse structure is operationally useful: training only the discovered subnetwork recovers nearly the same performance as full \textsc{OPD}. However, the sparsity-inducing SGD optimizer underperforms AdamW in our optimizer ablation, likely because dense teacher supervision preserves heterogeneous coordinate-wise gradient scales where AdamW's adaptive scaling remains useful. On geometry, the updates are numerically full-rank but spectrally concentrated; they lie mostly away from the principal singular subspaces of the source weights and fall disproportionately on coordinates where the source weights are close to zero. These findings suggest that dense teacher supervision does not turn \textsc{OPD} into ordinary dense parameter rewriting; instead, \textsc{OPD} retains important geometric signatures of on-policy post-training.