Search papers, labs, and topics across Lattice.
This study investigates the impact of data repetition in the pretraining of large language models (LLMs) as their token budgets scale with model size. The authors reveal that, counterintuitively, the optimal repetition count for high-quality domain data increases with model size, while the relationship between unique domain data and optimal repetition is weak. Their findings indicate that tuning repetition counts on smaller models can effectively inform larger model training, potentially enhancing performance without leading to overfitting.
Surprisingly, larger LLMs benefit from increased repetition of high-quality domain data, challenging conventional wisdom about data diversity in training.
As large language models scale, their training-token budgets must also increase to maintain an appropriate tokens-per-parameter ratio (\(TPP\)). However, high-quality domain data is much harder to scale than general web data. As model size and the training-token budget increase, its fraction in the training mixture tends to decrease. Repeating the available high-quality data provides an effective way to counteract this dilution, but excessive repetition may lead to overfitting. We study this trade-off under practical LLM scaling, where the training-token budget grows proportionally with model size. For a fixed domain, we first find that, surprisingly at a fixed \(TPP\), the optimal repetition count mildly increases with model size. Across different domains, we find that the optimal repetition count is strongly negatively correlated with the final validation loss of a domain: domains with lower loss can generally benefit from more repetitions. In contrast, the amount of unique domain data is only weakly related to the optimal repetition count. These findings suggest that repetition counts tuned on smaller proxy models with the same \(TPP\) can provide a practical estimate for larger models.
Tsinghua AI