Search papers, labs, and topics across Lattice.
This paper addresses the reliability issue in Mixture-of-Experts (MoE) transformers where imperfect routing can send tokens to undertrained experts, by introducing Distributionally Robust MoE Training (DRMoET). DRMoET optimizes for high-loss routing outcomes by updating expert distributions via an entropy-regularized softmax on EMA-smoothed, activation-weighted losses, thereby strengthening plausible non-top routing paths. Experiments with the FLAME-MoE recipe at up to 10.3B parameters demonstrate that DRMoET improves downstream performance over standard MoE and auxiliary-loss-free balancing, achieving a 0.6767 seven-task average compared to 0.6625 for standard FLAME-MoE.
Routing robustness, not just load balancing, is the key to reliable sparse MoE scaling, with DRMoET boosting performance by 1.4% on a 10.3B parameter model.
Mixture-of-Experts (MoE) transformers scale capacity by activating only a few experts per token, but this sparsity creates a hidden reliability problem: when routing is imperfect, load-balanced models may send tokens to experts that are insufficiently trained for the assigned inputs. We propose Distributionally Robust MoE Training (DRMoET), a drop-in objective that treats layer-wise experts as endogenous robustness groups and optimizes high-loss routing outcomes rather than merely equalizing traffic. DRMoET updates a per-layer expert distribution by an entropy-regularized softmax rule on EMA-smoothed, activation-weighted expert losses, strengthening plausible non-top routing paths while preserving standard MoE computation. Under the FLAME-MoE recipe at 746M-total and 10.3B-total scales, DRMoET improves downstream averages over both standard FLAME-MoE and auxiliary-loss-free balancing. At 10.3B total parameters and 67B training tokens, DRMoET improves the seven-task average from 0.6625 to 0.6767, while the auxiliary-loss-free baseline achieves 0.6431. Mechanistic analyses show lower expert-loss variance with nearly unchanged mean loss, 4.3% lower excess loss under forced mid-$k$ misrouting, and improved domain-expert specialization. These results position routing robustness-not only utilization balance-as a practical objective for reliable sparse MoE scaling. Project page and code are available at: https://drmoet.github.io/.
