Search papers, labs, and topics across Lattice.
This paper introduces elbow-based routing as a novel, training-free method for dynamically selecting experts in Mixture-of-Experts (MoE) models during inference. By identifying an elbow point in the sorted router probability distribution, the approach allows for a more efficient allocation of compute resources, activating only the most relevant experts per token. Empirical results show that this method achieves an average latency reduction of 5.3% while maintaining accuracy across multiple benchmarks, enhancing the efficiency of MoE models without compromising performance.
Elbow-based routing can cut inference latency by over 5% in MoE models while preserving accuracy, revolutionizing expert selection efficiency.
Mixture-of-Experts (MoE) models enable model scaling while maintaining low inference-time compute by activating only a subset of experts per token. However, conventional routing relies on a fixed top-k selection, forcing the model to spend the same compute regardless of how many experts are relevant. We introduce elbow-based routing, a training-free inference-time modification that dynamically adjusts the number of experts on a per-token basis. Our method examines the sorted router probability distribution and identifies an elbow point that separates high- and low-probability experts. We find that most router distributions exhibit clear inflection points suitable for this strategy, and we show both theoretically and empirically that elbow-based routing preserves expert load balance. Experiments on a state-of-the-art MoE model demonstrate an average latency reduction of 5.3% while maintaining accuracy across six benchmarks.
