Search papers, labs, and topics across Lattice.
This paper introduces KletterMix, a high-quality German pretraining corpus created by translating a state-of-the-art English dataset while maintaining document integrity and diversity. The authors validate the dataset's quality through comprehensive analyses and demonstrate its effectiveness in enhancing model performance on German-language tasks. Results indicate that models trained on KletterMix outperform those trained on existing German corpora, highlighting the potential of well-curated translated data in improving language model training.
Translated datasets can significantly elevate the quality of German language models, with KletterMix showing measurable performance gains over existing resources.
High-quality pretraining data is a central ingredient in modern language models, but German-language resources remain far less developed than their English counterparts: they are often smaller, less carefully curated, weakly documented, and rarely validated through controlled training experiments. We introduce KletterMix, a high-quality German corpus for language model pretraining and annealing, designed as a reusable dataset artifact for the natural language processing and modeling community. KletterMix is built by translating a state-of-the-art English pretraining corpus into German while preserving document boundaries, metadata, source structure, and topical diversity. This construction yields a German corpus with the scale and diversity of a modern pretraining dataset, while enabling direct comparison to its English source. We document the dataset through a broad set of corpus-level analyses, including translation quality, document length distributions, topic coverage, source composition, and geographic metadata. Using COMETKiwi, we show that the translated documents achieve strong quality across diverse domains, suggesting that careful translation can preserve much of the semantic and stylistic richness of the original corpus. Beyond dataset construction, we evaluate KletterMix as training data. Through controlled pretraining and annealing ablations against established German corpora, we show that models trained on KletterMix achieve measurable improvements on German-language downstream evaluations. These results demonstrate that carefully curated translated data can substantially strengthen the German pretraining data ecosystem.
