Track genomics, neuroscience and molecular biology.
Search papers, labs, and topics across Lattice.
This paper systematically investigates parameter-efficient fine-tuning (PEFT) strategies for Mixture-of-Experts (MoE) Large Language Models (LLMs), addressing a critical gap in adapting these efficient architectures. Through extensive experiments across three MoE backbones, the study reveals that adapting MoE experts alone yields superior downstream performance with strong general capability retention. Key findings include the superiority of native expert-weight updates over bypass adapters, the effectiveness of LoRA and DoRA, and that zero load-balancing coefficients are optimal for downstream tasks.
Adapting MoE experts directly, not through bypass adapters, unlocks the best downstream performance with minimal general capability loss.
Parameter-efficient fine-tuning (PEFT) methods are critical for adapting large models under resource-constrained settings. However, as mixture-of-experts (MoE) architectures, which offer efficient inference at massive parameter scales, gain increasing prevalence in the community, a critical gap arising from the lack of systematic recipes for applying PEFT to MoE models. To narrow this gap, we conduct a large-scale systematic study that investigates a set of essential design recipes, including which modules should be adapted, which granularity should be used to adapt MoE modules, which PEFT method to use, and to what extent should the load-balancing objective be used, across three MoE backbones on downstream tasks and general knowledge tasks. Our experiments yield several key insights. First, adapting MoE experts alone achieves the highest downstream performance with competitive general capability retention under a comparable parameter budget. Second, native expert-weight updates offer a better downstream--retention trade-off than adapters that bypass experts or entire MoE modules. Third, LoRA and DoRA together define the strongest operating region for high downstream accuracy among the evaluated PEFT methods. Fourth, no nonzero load-balancing coefficient Pareto-dominates the zero baseline. We finally evaluate a weight-level redesign of MoE$^2$-LoRA on Qwen3. Moving its module-level bypass pool into native expert weights improves all four upstream endpoints by 2.4 to 17.9 points with 17% fewer trainable parameters and a 3.2-point average downstream cost.