Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates how Mixture-of-Experts (MoE) architectures impact capacity allocation and performance degradation when scaling multilingual models. By proposing and fitting a repetition-aware joint scaling law on extensive empirical data, the authors quantify the benefits of MoE in mitigating the performance penalty of adding languages. The key result shows that 32 experts can reduce the loss penalty of expanding from 12 to 200 languages by 83% compared to a dense model at matched active parameters and per-language tokens.
MoE models slash the performance cost of multilingual expansion by 83% when scaling from 12 to 200 languages, fundamentally altering the trade-offs in capacity allocation.
Multilingual models share capacity across languages, so adding languages can hurt performance on existing ones. Mixture-of-experts models expand this capacity with little additional per-token computation. Given this extra capacity, does adding experts reduce what each additional language costs? To answer this question, we propose a repetition-aware joint scaling law relating number of active parameters, training tokens per language, number of experts, and language coverage. Fitted on 26,300 per-language loss observations from 64 pretrained configurations, our law reduces prediction error on held-out configurations by 28% relative to the strongest evaluated baseline. At matched number of active parameters and training tokens per language, 32 experts reduce the loss penalty of expanding from 12 to 200 languages by 83% compared with a dense model. Our law further shows that broader language coverage favors more experts across a wider range of compute budgets, guiding capacity allocation as multilingual models scale.