Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates how multi-teacher on-policy distillation (MOPD) influences student model parameter updates by analyzing gradient dynamics and learning curves of Qwen3-1.7B trained with four RL-derived teachers. It reveals that loss averaging implicitly weights teacher contributions, Adam's first moment smooths parameter update discrepancies, and BF16 precision can obscure subtle weight changes. Crucially, the choice of averaging strategy significantly impacts downstream task performance, demonstrating that MOPD's effectiveness is sensitive to these underlying mechanisms.
Teacher averaging in MOPD, not just raw gradients, dictates student learning, with Adam's momentum and BF16 precision masking significant parameter shifts.
Multi-teacher on-policy distillation (MOPD) aims to combine the strengths of RL-trained teachers in a single student, but how teacher signals affect parameter changes remains underexplored. We study Qwen3-1.7B with four domain teachers trained with RL from the same initialization as the student, comparing gradients, optimizer updates, and task learning curves, with additional SmolLM3-3B diagnostics. We find that several factors influence teacher signals. First, loss averaging implicitly weights responses: token averaging favors longer responses, and equalizing domain contributions retains this weighting within domains. Second, Adam's first moment reduces differences in parameter updates: the cosine similarity is 0.83 between teachers and 0.96 between averaging rules, despite differences in raw gradients. Third, BF16 rounding hides small changes: about 97\% of FP32 master weights differ from initialization, but only 7--11\% of BF16 weights do. Finally, the top-64 intersection KL gradient closely matches Qwen's full-vocabulary gradient, but the effect on task performance depends on averaging: mathematics accuracy is 2.6 points higher than with sampled-token policy-gradient (PG) under response averaging and 2.1 points lower under global token averaging.