Search papers, labs, and topics across Lattice.
This paper investigates whether domain-specific experts emerge within Mixture of Experts (MoE) based Large Language Models (LLMs). Through evaluations of ten MoE-based LLMs (3.8B to 120B parameters), the authors find empirical evidence supporting the existence of such domain-specific experts. Based on this finding, they propose Domain Steering Mixture of Experts (DSMoE), a training-free inference method that leverages these experts to improve performance on both target and non-target domains.
Unlocking hidden domain expertise within MoE models can boost performance without any extra training or inference costs.
In the era of Large Language Models (LLMs), the Mixture of Experts (MoE) architecture has emerged as an effective approach for training extremely large models with improved computational efficiency. This success builds upon extensive prior research aimed at enhancing expert specialization in MoE-based LLMs. However, the nature of such specializations and how they can be systematically interpreted remain open research challenges. In this work, we investigate this gap by posing a fundamental question: \textit{Do domain-specific experts exist in MoE-based LLMs?} To answer the question, we evaluate ten advanced MoE-based LLMs ranging from 3.8B to 120B parameters and provide empirical evidence for the existence of domain-specific experts. Building on this finding, we propose \textbf{Domain Steering Mixture of Experts (DSMoE)}, a training-free framework that introduces zero additional inference cost and outperforms both well-trained MoE-based LLMs and strong baselines, including Supervised Fine-Tuning (SFT). Experiments on four advanced open-source MoE-based LLMs across both target and non-target domains demonstrate that our method achieves strong performance and robust generalization without increasing inference cost or requiring additional retraining. Our implementation is publicly available at https://github.com/giangdip2410/Domain-specific-Experts.
