Search papers, labs, and topics across Lattice.
This paper introduces SoftMoE, a novel approach that replaces the discrete top-$k$ routing in Sparse Mixture-of-Experts (MoE) architectures with a differentiable soft top-$k$ LapSum relaxation, enabling gradient-based optimization of expert routing. By parameterizing the mean number of active experts and imposing a global budget constraint, SoftMoE allows for more efficient expert allocation across layers, resulting in significant computational savings. The method demonstrates performance on par with or superior to traditional sparse MoE models in language tasks while activating fewer experts, highlighting a non-uniform allocation strategy that favors later layers.
SoftMoE achieves comparable or superior performance to traditional sparse MoE while activating significantly fewer experts, revolutionizing expert allocation in LLMs.
Sparse Mixture-of-Experts (MoE) architectures enable scaling LLM parameters under a fixed inference budget by activating only a small subset of experts via top-$k$ routing. While this preserves causality and suits autoregressive language models, the discrete top-$k$ operator is not differentiable, forcing a fixed number of active experts per input and resulting in inefficient use of computation. We propose SoftMoE, which replaces discrete routing with a truncated soft top-$k$ LapSum relaxation, allowing gradient-based optimization of expert routing. We further parameterize the mean number of active experts per layer and impose a global budget constraint, enabling the model to learn how to allocate expert capacity across layers. SoftMoE remains fully compatible with autoregressive modeling and achieves performance comparable to or better than sparse MoE on language modeling and downstream tasks, while activating significantly fewer experts. Notably, the learned allocation is highly non-uniform, with later layers activating more experts. The source code is publicly available$^\dagger$.
