Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
This paper introduces Nash-MOPD, an adaptive domain scheduling algorithm for multi-teacher on-policy distillation that addresses inflexible allocation and under-training of slow-improving domains by framing domain sampling as a local minimax game. The equilibrium strategy maximizes the minimum predicted proportional reduction in reverse-KL divergence across domains, effectively allocating more optimization steps to domains requiring more examples for comparable improvement. Nash-MOPD demonstrates superior performance over existing baselines on mathematics and code accuracy metrics, achieving competitive or leading scores on multiple benchmarks.
A novel minimax game formulation for multi-teacher distillation dynamically allocates optimization budget to ensure equitable progress across domains, even those with slower learning rates.
Multi-teacher on-policy distillation (MOPD) aggregates the capabilities of multiple domain-expert teachers into a single student model. However, all domains in MOPD share the student's finite budget of optimization steps and rollouts. Fixed uses constant domain sampling ratios based on corpus sizes; by contrast, D$^3$-MOPD adjusts the ratios using the teacher-student reverse-KL divergence and its recent decrease. However, favoring large recent decreases can direct more training toward domains that already improve quickly, while domains that improve more slowly may receive too few examples. To address inflexible allocation and the risk of giving insufficient training to domains that improve more slowly, we propose Nash-MOPD, an adaptive domain scheduler based on a local minimax game using existing reverse-KL statistics. The equilibrium maximizes the smallest predicted proportional reduction under the local model and gives more prompts to domains that need more examples for comparable improvement. Consequently, Nash-MOPD responds to changing learning progress and supports domains that improve more slowly without extra teacher queries or rollouts. At step 100, Nash-MOPD exceeds both baselines' final mathematics and code checkpoint scores on DeepSeek-R1-Distill-Qwen-1.5B and reaches Fixed's on Qwen3-8B-Base. Furthermore, it leads eight of nine reported metrics and is best or tied best on all six mathematics and code accuracy metrics, respectively.