Search papers, labs, and topics across Lattice.
This paper investigates the optimal allocation of compute between general pretraining and specialized continued pretraining for language models in multi-domain settings. They introduce a method to predict the loss of a model based on its size and the number of general pretraining and specialization tokens, using scaling laws to determine the optimal compute split. Experiments demonstrate consistent performance improvements on common sense knowledge and reasoning benchmarks by optimizing the pretraining/specialization compute allocation.
Stop guessing how much to pretrain vs. specialize your language model – scaling laws can now tell you the optimal compute allocation for maximizing performance on downstream tasks.
Language models achieve impressive performance on a variety of knowledge, language, and reasoning tasks due to the scale and diversity of pretraining data available. The standard training recipe is a two-stage paradigm: pretraining first on the full corpus of data followed by specialization on a subset of high quality, specialized data from the full corpus. In the multi-domain setting, this involves continued pretraining of multiple models on each specialized domain, referred to as split model training. We propose a method for pretraining multiple models independently over a general pretraining corpus, and determining the optimal compute allocation between pretraining and continued pretraining using scaling laws. Our approach accurately predicts the loss of a model of size N with D pretraining and D'specialization tokens, and extrapolates to larger model sizes and number of tokens. Applied to language model training, our approach improves performance consistently across common sense knowledge and reasoning benchmarks across different model sizes and compute budgets.
