Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
Multi-teacher on-policy distillation often fails to retain specialized capabilities because high-variance feedback from domains like instruction-following dominates gradient updates, severely diluting signals from narrower domains like mathematics. To address this gradient imbalance across Qwen3.5 models, the authors establish Domain-Normalized MOPD (DN-MOPD), which rescales each specialist's token-level supervisory signal inversely by its empirically measured variance. This normalization consistently outperforms standard routing across six benchmarks and three model scales, recovering the majority of lost reasoning performance primarily by down-weighting over-dispersed instruction-following feedback.
Multi-teacher distillation fails not because of bad routing, but because high-variance instruction feedback quietly drowns out precision domains like mathematics during shared student updates.
Reinforcement learning can turn one language model into several specialists, each excellent at a single skill such as mathematics, coding or following instructions, but users need one model with all of these skills. Multi-teacher on-policy distillation (MOPD) merges them by letting the specialists teach one student: the student answers each prompt, and the specialist for that prompt's domain gives feedback on every token. This routing decides which specialist teaches, but not how strongly its feedback moves the shared student. In Qwen3.5 models at three sizes, we find that MOPD's student does not beat one taught by the best single specialist and gains little of the mathematics specialist's advantage. The feedback is unbalanced: instruction-following feedback is several times more spread out than mathematics feedback and dominates the student's updates. We propose Domain-Normalized MOPD (DN-MOPD), which keeps the routing and rescales each domain's feedback by its measured spread. On six public benchmarks, DN-MOPD improves the average score over MOPD at every size, across three random seeds and under two answer-length limits, and recovers most of the lost mathematics gain. Controls with fixed domain weights show that the gain comes mainly from turning down instruction-following feedback rather than turning up mathematics alone, and that fixed weights close to those DN-MOPD measures perform comparably. Combining specialists therefore requires deciding not only which one teaches, but also how strongly its feedback counts.