Search papers, labs, and topics across Lattice.
This paper introduces FEPLB, a novel load balancing scheme for Mixture-of-Experts (MoE) training that leverages the NVIDIA Hopper architecture's NVLink Copy Engine for nearly free intra-node communication. By using the Copy Engine to redistribute tokens and weights within the NVLink domain, FEPLB minimizes communication overhead associated with dynamic expert scheduling. Experiments on GLM-5 demonstrate that FEPLB reduces token and GEMM stragglers by 50-70% without measurable EP communication overhead, achieving up to 2x lower token straggler than FasterMoE at EP=8.
Hopper's idle NVLink Copy Engine can be turned into a nearly free communication channel for MoE load balancing, slashing token stragglers by up to 70% without impacting existing parallelism.
Fine-grained, per-micro-batch load balancing is essential for efficient Mixture-of-Experts (MoE) training, yet every prior dynamic scheduling scheme pays for it with extra communication that is hard to hide. Especially on modern bulk-transfer backends such as DeepEP. We make a simple but consequential observation: on the NVIDIA Hopper architecture the NVLink Copy Engine can move data between intra-node GPUs without consuming any SM cycles, effectively providing a nearly free communication channel that runs in parallel with compute kernels. FEPLB turns this idle hardware into a new parallel dimension for MoE load rebalancing. Its Two-Phase Dispatch first routes tokens across nodes via the standard EP backend, then redistributes dynamic-expert tokens and weights within the NVLink domain through the Copy Engine at nearly zero cost, while a lightweight CPU scheduler runs concurrently with static expert computation. Because FEPLB uses only Copy Engine and CPU that are orthogonal to those consumed by EP and PP, it coexists with existing parallel strategies without reconfiguration. On GLM-5's MoE layers (128 experts, no auxiliary loss, up to 16 H100 GPUs), FEPLB reduces the token straggler by 51-70% and the GEMM straggler by 50-68% with no measurable EP communication overhead. Its advantage grows with the EP degree: at EP=8, it achieves 2x lower token straggler than FasterMoE.
