Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors develop SYNTH, an open-source, fully synthetic dataset derived from ~59k Wikipedia seed articles that collapses traditional pre-, mid-, and post-training into a unified, single-stage training pipeline. While frontier labs rely on proprietary synthetic mixes to inject reasoning and bypass the noise of raw web crawls, open-source understanding of synthetic data scaling and skill acquisition has remained severely constrained. Evaluated across scales up to a 13B/1B-active MoE, models trained entirely on SYNTH outperform filtered web baselines at iso-compute and match open-weight models while requiring 10–140× fewer tokens.
Collapsing pre-, mid-, and post-training into a single fully synthetic run matches open-weight model performance while slashing required training tokens by up to 140×.
Current pre-training datasets are derived from web crawls, with all their issues, and were not designed to support mid- and post-training pipelines--for instance, they contain little explicit reasoning. Thus, many frontier labs have begun to develop their own internal datasets, starting from state-of-the-art models, to augment their pre-training data mix, eg, with reasoning traces to address cold-start problems. While demonstratively effective, none of these datasets are public, and the effect of this so-called synthetic data on knowledge and skill acquisition of language models, including small ones, remains poorly understood. We present SYNTH, the first open-source synthetic corpus derived from 58,698 Wikipedia articles that collapses pre-, mid-, and post-training into a single training stage via structured amplification of curated encyclopedic seeds. We evaluate SYNTH by training a suite of models: a 56M tiny model (Monad), 0.3B-0.6B dense models (Baguettotron), and a 13B / 1B-active MoE. At iso-compute, SYNTH outperforms filtered web data, and our models remain competitive with similarly-sized open-weight baselines. Because SYNTH is back-translated from grounded passages, SYNTH-trained models achieve high factual precision despite 10-140x fewer training tokens, with memorization targeted by the seed corpus. These results show that synthetic datasets, including our SYNTH dataset, are capable of producing competitive generalist models from a fraction of the training data, enabling rapid iteration as the frontier advances. These findings open up possibilities for both generalist models with significantly increased data efficiency, as well as domain-specific models where no instruction or conversational data is available. Finally, we publicly release our SYNTH dataset and the suite of Baguettotron models under a permissive license, thus supporting open-source language model development.
TU Munich