Search papers, labs, and topics across Lattice.
This paper introduces MultiSynt/MT, a synthetic parallel corpus containing approximately 4.8 trillion tokens across 36 European languages, generated by translating 100 billion high-quality tokens using advanced translation systems. The significance lies in its ability to provide a vast multilingual pre-training resource, particularly for medium- and lower-resource languages, which has been a barrier in developing effective multilingual LLMs. Remarkably, LLMs trained on this corpus achieve performance comparable to native data benchmarks while utilizing 72% fewer tokens and outperforming them by 15% at a matched training budget.
LLMs trained on a synthetic corpus can outperform native data benchmarks while using significantly fewer tokens, challenging the assumption that more data always leads to better performance.
Open web-scale pre-training corpora remain concentrated in English, limiting multilingual LLM development. We introduce MultiSynt/MT, an open synthetic parallel corpus with approximately 4.8 trillion target-language tokens across 36 European languages, produced by translating 100 billion high-quality Nemotron-CC tokens with Tower+ and OPUS-MT/HPLT-MT systems. For many medium- and lower-resource European languages, this is the largest openly available pre-training resource. On a broad multilingual benchmark suite, reference LLMs trained on MultiSynt/MT reach the final score of HPLT 2.0, a native-data baseline, using roughly 72% fewer pre-training tokens, and outperform it by approximately 15% relative at a matched 100B-token training budget. Our analyses also identify evaluation blind spots: standard multiple-choice benchmarks miss translation-quality differences that a fluency-sensitive LLM-as-judge evaluation cleanly recovers on the trained LLMs (with no fluency deficit in MultiSynt itself), and Norwegian idiomatic and culturally grounded tasks remain better served by native data. We release the corpus, including row-aligned translations from multiple systems, to support controlled research on multilingual pre-training data and evaluation.
