Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
This paper introduces training-free on-policy distillation (\method), a method that consolidates specialist models into a single student model using only forward passes, eliminating the need for gradient steps or teacher rollouts during distillation. The core insight is that specialist capabilities can be aggregated by solving a layer-wise least-squares problem derived from input second moments and specialist outputs, achieving a generalist model that surpasses individual specialists in accuracy. Experiments demonstrate that this approach, when applied to mathematics and code benchmarks, yields a generalist model that retains significant performance gains from its specialist predecessors with minimal computational cost.
Consolidating specialist LLMs into a single, more accurate generalist is possible with *zero* gradient updates, relying solely on forward passes and input statistics.
On-policy distillation consolidates specialist models into one student at the price of rollouts, teacher passes and gradient steps for every teacher. We show that the consolidation does not need training: with forward passes only, the capabilities of separately tuned specialists can be aggregated into one model that is, on average, more accurate than either specialist alone. When specialists share a base model, distillation has an exact layer-wise solution: each linear layer of the fused model must reproduce every specialist's output on the inputs that specialist receives while reading its own answers, a least-squares problem whose solution needs only the second moments of those inputs. One forward pass over a few hundred rollouts provides them. We call this training-free on-policy distillation (\method). What the rollouts contain matters more than the solver: answers truncated in repetition loops are few among a mathematics specialist's rollouts but carry a large share of their tokens, and removing them lifts the fused model on AIME25 from task arithmetic's level to the specialist's at matched precision. On Olmo~3 7B, a few GPU-hours of forward computation per specialist yield a generalist that keeps most of each specialist's gain on each of seven mathematics and code benchmarks.
Tsinghua AI