Search papers, labs, and topics across Lattice.
Marco-MoE, a family of multilingual Mixture-of-Experts models, achieves state-of-the-art performance-to-compute ratio by activating only 5% of parameters and leveraging upcycling from dense models during pre-training on 5T tokens. The resulting models outperform similarly sized competitors on English and multilingual benchmarks, and instruct-tuned variants surpass models with 3-14x more activated parameters. Analysis reveals structured expert activation patterns across related languages, while maintaining specialization for isolated languages, enabling scalable language expansion.
Multilingual MoEs can achieve best-in-class performance-to-compute ratios, even with extreme sparsity, by strategically upcycling from dense models and exhibiting structured expert activation patterns across languages.
We present Marco-MoE, a suite of fully open multilingual sparse Mixture-of-Experts (MoE) models. Marco-MoE features a highly sparse design in which only around 5\% of the total parameters are activated per input token. This extreme sparsity, combined with upcycling from dense models, enables efficient pre-training on 5T tokens. Our models surpass similarly-sized competitors on English and multilingual benchmarks, achieving a best-in-class performance-to-compute ratio. We further post-train these models to create Marco-MoE-\textsc{Instruct} variants, which surpass the performance of competing models possessing $3$--$14\times$ more activated parameters. Our analysis reveals that Marco-MoE learns structured expert activation patterns shared across related languages, while maintaining highly specialized utilization for linguistically isolated ones. We further show that Marco-MoE allows for scalable language expansion without the interference typical of dense models. To support the community, we disclose our full training datasets, recipes, and model weights.
DAMO