Search papers, labs, and topics across Lattice.
This paper introduces EPnG, an adaptive prune-and-grow framework designed for parameter-efficient fine-tuning of Mixture-of-Experts (MoE) models. By reallocating LoRA capacity based on expert importance derived from router gate probabilities, EPnG optimally prunes under-utilized experts and expands high-importance ones, all while maintaining a fixed parameter budget. The method outperforms traditional LoRA approaches, achieving comparable performance to full fine-tuning with a significantly reduced number of parameter updates (0.55%-0.72%), showcasing a more effective strategy for MoE fine-tuning.
EPnG achieves up to 180x fewer parameter updates while matching the performance of full fine-tuning in MoE models.
Mixture-of-Experts (MoE) models scale efficiently but remain costly to adapt due to redundant experts and uniform parameter allocation. Existing parameter-efficient fine-tuning (PEFT) methods such as LoRA ignore MoE routing dynamics, leading to suboptimal resource use. We propose EPnG, an adaptive prune-and-grow framework that reallocates LoRA capacity based on expert importance derived from router gate probabilities. EPnG prunes under-utilized experts and expands high-importance experts via rank growth with orthogonal initialization, while maintaining a fixed parameter budget. Across OLMoE and Qwen1.5-MoE, EPnG consistently outperforms LoRA under the same budget and achieves performance comparable to full fine-tuning while updating only 0.55%-0.72% of parameters (up to 140x-180x fewer). These results demonstrate that aligning PEFT with MoE routing yields a more effective and scalable fine-tuning strategy.
