Search papers, labs, and topics across Lattice.
The paper introduces Replicate-and-Quantize (R&Q), a novel inference-time method to address load imbalance in Sparse Mixture-of-Experts (SMoE) models without retraining. R&Q replicates heavily loaded experts and quantizes less utilized ones, maintaining the original memory footprint while increasing parallel processing capacity. Experiments demonstrate that R&Q reduces load imbalance by up to 1.4x while preserving accuracy within +/-0.6% across various SMoE models and benchmarks.
Solve SMoE load balancing at inference time without retraining by replicating heavily used experts and quantizing underutilized ones, achieving up to 1.4x imbalance reduction.
Sparse Mixture-of-Experts (SMoE) architectures are increasingly used to scale large language models efficiently, delivering strong accuracy under fixed compute budgets. However, SMoE models often suffer from severe load imbalance across experts, where a small subset of experts receives most tokens while others are underutilized. Prior work has focused mainly on training-time solutions such as routing regularization or auxiliary losses, leaving inference-time behavior, which is critical for deployment, less explored. We present a systematic analysis of expert routing during inference and identify three findings: (i) load imbalance persists and worsens with larger batch sizes, (ii) selection frequency does not reliably reflect expert importance, and (iii) overall expert workload and importance can be estimated using a small calibration set. These insights motivate inference-time mechanisms that rebalance workloads without retraining or router modification. We propose Replicate-and-Quantize (R&Q), a training-free and near-lossless framework for dynamic workload rebalancing. In each layer, heavy-hitter experts are replicated to increase parallel capacity, while less critical experts and replicas are quantized to remain within the original memory budget. We also introduce a Load-Imbalance Score (LIS) to measure routing skew by comparing heavy-hitter load to an equal allocation baseline. Experiments across representative SMoE models and benchmarks show up to 1.4x reduction in imbalance with accuracy maintained within +/-0.6%, enabling more predictable and efficient inference.
