Search papers, labs, and topics across Lattice.
This study investigates the impact of organizing synthetic textbook data into coherent book-level documents on language model performance during mid-training. By employing a scalable synthesis pipeline that clusters related content and assembles it into structured textbooks, the authors generated 686K textbooks across various disciplines, leading to an average performance improvement of +1.09 on downstream tasks. Controlled experiments further reveal that the benefits of this structured approach cannot be attributed solely to document length or independent section treatment, underscoring the significance of hierarchical organization in synthetic data design.
Book-level organization of synthetic data boosts language model performance by over 1% compared to traditional methods, highlighting the power of structured content.
Synthetic textbook data has improved language model pre-training, but prior work largely treats the benefit as a property of generated content or local rewriting style. We study a different factor: whether related content is organized into coherent book-level documents. We contribute both a scalable synthesis pipeline and controlled evidence that this organization matters. The pipeline retrieves source material from a pre-training corpus, clusters it into topical units, plans hierarchical tables of contents, and assembles source-grounded sections into complete books (our Full setting), yielding 686K textbooks (32B tokens) across 15,000+ disciplines. Replacing natural books in a mid-training mix with this corpus improves downstream performance by +1.09 on average. Controlled comparisons then disentangle the relevant design factors. A content-matched Split condition holds generated text and tokens fixed but treats each section as an independent document; Full's +1.02 mean gain isolates document packaging. A length-matched RandomConcat control that joins sections from different books remains below Full, ruling out document length alone. A retrieval-pool-matched Rephrase condition independently rewrites individual retrieved documents under the same audience-by-style scheme, without clustering, TOC planning, or book assembly; Full's +1.17 gain demonstrates the value of structured synthesis. On Llama3-8B, Full likewise outperforms both RandomConcat and Natural Books, supporting book-level organization as a useful axis for synthetic pre-training data design.
