Search papers, labs, and topics across Lattice.
This paper introduces Path-Aligned Decompression Distillation (PADD), a novel framework that enables mixture-of-experts (MoE) students to learn from dense teacher models without explicit routing, while simultaneously optimizing routing policies. The method is structured into four stages across two phases, focusing on initializing diverse expert functionality and employing adaptive distillation techniques to enhance learning efficiency. Experimental results show that PADD significantly outperforms strong baselines on mathematical reasoning tasks, allowing MoE students to achieve or exceed the performance of their dense teachers at the same inference cost.
MoE students can not only match but often surpass their dense teacher counterparts using a novel distillation framework that optimizes routing policies without explicit guidance.
As large language models (LLMs) continue to scale, it becomes increasingly challenging to grow model capacity under fixed computation budgets. We propose Path-Aligned Decompression Distillation (PADD), a framework for distilling knowledge from dense teachers without explicit routing into mixture-of-experts (MoE) students while learning high-quality routing policies. PADD organizes knowledge distillation into four stages in two phases: an initialization phase (Stage I) that builds diverse functionality in the student's experts through teacher neuron clustering and student-expert warmup, and a training phase (Stages II--IV) that integrates online adaptive distillation, path-refined policy optimization, and reward-augmented load balancing in a single training pipeline. Experiments on mathematical reasoning benchmarks demonstrate that PADD yields substantial gains over strong baselines at the same inference cost and that the MoE student can match or surpass its dense teacher. They also demonstrate effective teacher-to-student knowledge distillation and stable routing behavior.
