Search papers, labs, and topics across Lattice.
This paper introduces D$^3$-MOPD, an adaptive dynamic domain scheduling method for multi-teacher on-policy distillation that optimizes the domain mixture during training based on real-time performance metrics. By leveraging the reverse-KL divergence signals generated during training, the method dynamically adjusts the sampling ratios of different domains, allowing for more efficient use of computational resources and improved training outcomes. The results show that D$^3$-MOPD significantly narrows the performance gap between the student model and its teachers, achieving 97% of the average teacher performance with a threefold reduction in rollout steps compared to traditional methods.
Closing 97% of the performance gap between student and teacher models while cutting rollout steps by nearly three times reveals the power of adaptive domain scheduling in multi-teacher distillation.
Multi-teacher on-policy distillation (MOPD) distills several domain-expert teachers into a single student by minimizing per-domain reverse-KL divergence on the student's own rollouts. Existing approaches typically fix the per-domain data mixture before training, overlooking the fact that different domains converge at substantially different rates: some plateau early while others continue to improve throughout the training budget. A fixed mixture therefore wastes compute on fast-converging domains and undertrains slower-converging ones. To address this, we propose D$^3$-MOPD (Dynamic Domain ScheDuling for MOPD), a zero-overhead scheduler that repurposes the per-domain reverse-KL signal already produced during training to adapt the domain mixture online. Running asynchronously outside the training process, an off-process watcher periodically tracks each domain's KL trajectory, estimates remaining headroom and current improvement rate, and accordingly adjusts the domain sampling ratios without altering the core training loop. Our D$^3$-MOPD scales naturally to arbitrary numbers of domains, and the expected benefit grows as more domains introduce more diverse convergence patterns for the scheduler to exploit. On a Qwen3.6-35B-A3B student distilled from four domain-expert teachers, D$^3$-MOPD closes 97% of the average student-to-teacher performance gap, compared with 63% for vanilla MOPD, reaches the same peak performance with an approximately 3$\times$ reduction in rollout steps, and surpasses the specialist teachers on three of seven benchmarks.
