Search papers, labs, and topics across Lattice.
This paper introduces ReaLB, a real-time load balancing method for multimodal Mixture-of-Experts (MMoE) inference that addresses the issue of skewed expert workloads caused by modality imbalances. ReaLB dynamically adjusts the computation precision of MoE experts at runtime on a per-expert-parallelism-rank basis, assigning lower precision to vision-heavy experts to improve execution efficiency. Experiments on MMoE models demonstrate a 1.29x layer-level speedup with minimal accuracy loss (within 1.2%).
ReaLB achieves 1.29x faster multimodal MoE inference by dynamically adjusting expert precision, proving that real-time adaptation can overcome modality-induced load imbalances.
Mixture-of-Experts (MoE) architectures are widely used in modern large language models and multimodal models. However, inference efficiency is often limited by highly dynamic and skewed expert workloads across different modalities. During the prefill stage with large batch sizes, vision tokens frequently dominate the input sequences. Under expert parallelism (EP), this leads to severe load imbalance, where a subset of devices becomes overloaded, reducing overall system throughput. We propose ReaLB, a real-time load balancing method for multimodal MoE (MMoE) inference that introduces zero scheduling overhead. ReaLB dynamically adjusts the computation precision of MoE experts at runtime on a per-EP-rank basis. For ranks dominated by vision-heavy experts, ReaLB assigns lower-precision computation to improve execution efficiency by exploiting FP4 Tensor Cores. ReaLB does not require redundant experts or additional memory allocation. Instead, it performs layer-wise expert precision transformation on the fly and hides the associated overhead within the dispatch phase before MoE computation. Experiments on representative MMoE models show that ReaLB achieves 1.29x layer-level speedup while limiting accuracy loss to within 1.2%.
