Search papers, labs, and topics across Lattice.
The authors investigated multilingual data curation across thirteen languages to address challenges in training high-quality multilingual models, finding that many performance regressions are due to data quality and composition issues rather than inherent limitations. They demonstrated that improving data quality in one language benefits others in bilingual experiments and that bespoke per-language curation yields even larger improvements. By applying curated multilingual allocations comprising under 8% of total tokens to a 20T-token pretraining corpus, they achieved competitive multilingual accuracy with significantly fewer training FLOPs, establishing a new Pareto frontier in multilingual performance versus compute.
Multilingual interference isn't a fundamental capacity limit, but a data problem: targeted curation of just 8% of your training data can yield 4-10x FLOPs savings.
Multilinguality is a core capability for modern foundation models, yet training high-quality multilingual models remains challenging due to uneven data availability across languages. A further challenge is the performance interference that can arise from joint multilingual training, commonly referred to as the "curse of multilinguality". We study multilingual data curation across thirteen languages and find that many reported regressions are not inherent to multilingual scaling but instead stem from correctable deficiencies in data quality and composition rather than fundamental capacity limits. In controlled bilingual experiments, improving data quality for any single language benefits others: curating English improves non-English performance in 12 of 13 languages, while curating non-English yields reciprocal improvements in English. Bespoke per-language curation produces substantially larger within-language improvements. Extending these findings to large-scale general-purpose training mixtures, we show that curated multilingual allocations comprising under 8% of total tokens remain remarkably effective. We operationalize this approach within an effort that produced a 20T-token pretraining corpus derived entirely from public sources. Models with 3B and 8B parameters trained on a 1T-token random subset achieve competitive multilingual accuracy with 4-10x fewer training FLOPs than strong public baselines, establishing a new Pareto frontier in multilingual performance versus compute. Moreover, these benefits extend to frontier model scale: the 20T-token corpus served as part of the pretraining dataset for Trinity Large (400B/A13B), which exhibits strong multilingual performance relative to its training FLOPs. These results show that targeted, per-language data curation mitigates multilingual interference and enables compute-efficient multilingual scaling.
