Search papers, labs, and topics across Lattice.
This paper introduces Multi-Teacher On-Policy Distillation (MOPD), a novel approach for integrating multiple capabilities into large language models (LLMs) during post-training. By employing specialized reinforcement learning (RL) for each domain to create a set of domain teachers and then distilling their knowledge into a student model through its own rollouts, MOPD effectively eliminates exposure bias and enhances optimization. The method outperforms existing techniques like Mix-RL and Off-Policy Finetune on the Qwen3-30B-A3B model, demonstrating its ability to retain nearly all capabilities of the individual teachers while allowing for independent development of domain-specific models.
MOPD achieves superior capability integration in LLMs by distilling knowledge from multiple RL teachers without losing performance, setting a new standard for post-training methods.
Modern large language models (LLMs) rely on reinforcement learning during post-training to push specific capabilities, yet integrating multiple capabilities into one model remains hard. Existing methods, such as Off-Policy Finetune and Mix-RL, are either inefficient or lose performance. In this work, we propose Multi-teacher On-Policy Distillation (MOPD), a post-training paradigm for combining the capabilities of multiple domain RL teachers: we first run per-domain specialised RL to obtain a set of domain teachers, then distill these teachers into the student on its own rollouts. This eliminates exposure bias and provides a dense optimization signal. On Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines, inheriting nearly all of each teacher's capability. MOPD also enables parallel, independent development of domain teachers, removing the cross-domain coupling typical of multi-domain post-training. MOPD has been deployed in the post-training of MiMo-V2-Flash, an industrial-scale frontier model, demonstrating its practical value for capability integration in frontier-scale LLMs.
