Search papers, labs, and topics across Lattice.
This paper introduces MoEGen, a novel framework for parameter-efficient fine-tuning (PEFT) that leverages a mixture-of-experts (MoE) approach to generate instance-specific adaptations without the need for storing multiple full LoRA experts. By utilizing small learnable vectors called expert codes, MoEGen routes inputs through a hypernetwork that produces tailored low-rank updates, effectively decoupling expert capacity from adapter storage. Experimental results demonstrate that MoEGen consistently outperforms existing static and MoE-based PEFT methods across various commonsense reasoning benchmarks and shows strong performance in specialized domains like medical and legal adaptation.
MoEGen achieves instance-specific adaptations without the storage burden of full LoRA experts, revolutionizing how we think about parameter-efficient fine-tuning.
Parameter-efficient fine-tuning (PEFT) enables efficient adaptation of large language models, but existing MoE-based PEFT methods typically improve capacity by storing multiple full LoRA experts, causing adapter storage to grow linearly with the number of experts and restricting adaptation to a fixed expert pool. We ask whether MoE-based PEFT can produce instance-specific adaptations without explicitly storing a separate LoRA module for each expert. To address this gap, we propose MoEGen, an adaptation framework that shifts MoE-based PEFT from expert selection to expert-conditioned parameter generation. Instead of storing each expert as a full LoRA adapter, MoEGen represents each expert as a small learnable vector, termed an expert code. It routes each input over these vectors and uses their weighted combination to condition a lightweight hypernetwork that generates input-specific low-rank updates. This design decouples expert capacity from adapter storage while enabling instance-conditioned adaptation. Experiments on eight commonsense reasoning benchmarks show consistent improvements over strong static and MoE-based PEFT baselines across three backbones. MoEGen also performs strongly in joint medical and legal-domain adaptation.
