Search papers, labs, and topics across Lattice.
This paper explores cross-lingual transfer learning for training quality classifiers to filter multilingual pretraining data, addressing the challenge of insufficient high-quality data in low-resource languages. They evaluate strategies like cross-lingual transfer, third quartile sampling (Q3), and retention rate tuning, training a 1B model on 103B tokens across multiple languages. Results show that multilingual pooling often surpasses monolingual baselines in rank stability and aggregate accuracy, particularly when combined with techniques like Q3 sampling or retention rate tuning for high-resource languages.
Multilingual data quality classifiers can outperform monolingual ones, but only with careful tuning of the decision boundary, challenging the assumption that scale alone guarantees improved filtering.
As Large Language Models (LLMs) scale, data curation has shifted from maximizing volume to optimizing the signal-to-noise ratio by performing quality filtering. However, for many languages, native high quality data is insufficient to train robust quality classifiers. This work investigates the idea that quality markers in embedding space may show cross-lingual consistency, which would allow high-resource languages to subsidize the filtering of low-resource ones. We evaluate various filtering strategies, including cross-lingual transfer, third quartile sampling (Q3), and retention rate tuning. Our results demonstrate that massive multilingual pooling frequently outperforms monolingual baselines in both rank stability and aggregate accuracy for a 1B model trained on 103B tokens, delivering gains for high resource languages (1.2% increase in aggregate normalized accuracy for French) and matching or exceeding monolingual baselines for low-resource languages. However, we find that scale alone does not guarantee stability. Furthermore, for high-resource languages like French, we show that refining the decision boundary through third quartile sampling (Q3) or tuning the retention rate is necessary to fully leverage the multilingual signal.
