Search papers, labs, and topics across Lattice.
This paper introduces a systematic framework for converting trained Mixture-of-Experts (MoE) models into fully dense architectures, addressing the memory limitations of MoE during deployment. By scoring, selecting, and grouping experts, then refining them through knowledge distillation, the authors evaluate various configurations and find that their diversity-aware scoring method significantly enhances performance. The results show that the MoE-to-dense conversion outperforms traditional dense-to-dense pruning by +6.3 percentage points in downstream accuracy, while also achieving faster training speeds.
Diversity-aware scoring transforms MoE models into dense architectures, boosting downstream accuracy by over 6% while speeding up training.
Mixture-of-Experts (MoE) is now the dominant architecture for frontier language models, yet it requires all expert parameters to be loaded in memory, making it less preferable for memory-constrained deployment. Existing compression methods reduce the number of experts but the output remains an MoE model with the same fundamental limitation. We present the first systematic framework for converting a trained MoE into a standard fully dense architecture: experts are scored, selected, and grouped, then concatenated into a dense FFN and refined by knowledge distillation from the MoE teacher. We evaluate 7 scoring, 5 grouping, and 2 magnitude scaling methods across a range of selected expert counts on Qwen3-30B-A3B, yielding 350 configurations. We find that the choice of scoring method is the most impactful, with our novel diversity-aware scoring consistently outperforming prior methods on Qwen3-30B-A3B, DeepSeek-V2-Lite, and GPT-OSS-20B. Under a controlled comparison at matched parameter count, MoE-to-dense outperforms dense-to-dense pruning by +6.3 pp in average downstream accuracy after ~4B-token distillation at 1.6x faster training wall-clock speed.
