Search papers, labs, and topics across Lattice.
SAMoRA improves multi-task learning in LLMs by addressing imprecise routing and uniform weight fusion in MoE-LoRA methods. It introduces a Semantic-Aware Router to align textual semantics with appropriate experts and a Task-Adaptive Scaling mechanism to dynamically adjust expert contributions based on task requirements. Experiments on multi-task benchmarks show SAMoRA outperforms state-of-the-art methods and exhibits strong task generalization.
Task-specific LLMs can be efficiently fine-tuned by explicitly routing inputs to LoRA experts based on semantic similarity, rather than relying on implicit or uniform weighting schemes.
The combination of Mixture-of-Experts (MoE) and Low-Rank Adaptation (LoRA) has shown significant potential for enhancing the multi-task learning capabilities of Large Language Models. However, existing methods face two primary challenges: (1)Imprecise Routing in the current MoE-LoRA method fails to explicitly match input semantics with expert capabilities, leading to weak expert specialization. (2)Uniform weight fusion strategies struggle to provide adaptive update strengths, overlooking the varying complexity of different tasks. To address these limitations, we propose SAMoRA (Semantic-Aware Mixture of LoRA Experts), a novel parameter-efficient fine-tuning framework tailored for task-adaptive learning. Specifically, A Semantic-Aware Router is proposed to explicitly align textual semantics with the most suitable experts for precise routing. A Task-Adaptive Scaling mechanism is designed to regulate expert contributions based on specific task requirements dynamically. In addition, a novel regularization objective is proposed to jointly promote expert specialization and effective scaling. Extensive experiments on multiple multi-task benchmarks demonstrate that SAMoRA significantly outperforms the state-of-the-art methods and holds excellent task generalization capabilities. Code is available at https://github.com/boyan-code/SAMoRA
