Search papers, labs, and topics across Lattice.
This paper introduces WebGraphMix, a novel framework for pretraining data selection that leverages structural centrality scores from the Common Crawl host-level web graph to optimize the mixture of central and peripheral documents. By hypothesizing that central hosts provide reusable abstractions while peripheral hosts contribute specialized knowledge, the authors demonstrate that a balanced mixture of these document types enhances model performance across various tasks. Their experiments reveal that a 1:1 mixture of central and peripheral documents leads to a significant performance improvement, achieving an average score of 41.4%, surpassing uniform sampling methods and highlighting the importance of web graph topology in data curation.
Balancing central and peripheral web documents boosts model performance by over 1.6%, proving that web graph topology is a critical factor in pretraining data selection.
The performance of modern language models depends critically on pretraining data composition. Yet existing data selection methods rely on auxiliary classifiers for document scoring or mixture optimization, adding computational overhead and dependence on labeled data. We propose WebGraphMix, a lightweight data selection framework that computes structural centrality scores over the Common Crawl host-level web graph and uses them to vary the proportion of central versus peripheral documents in the pretraining mixture. We hypothesize that central hosts expose models to reusable abstractions, while peripheral hosts encode specialized, long-tail knowledge. WebGraphMix computes centrality scores efficiently at web scale, requiring no model training, labeled data, or downstream supervision. We integrate WebGraphMix into the DataComp-LM pipeline and train models at 400M and 1B parameter scales with 8B and 28B tokens respectively, evaluating on 23 tasks ranging from factual knowledge to symbolic reasoning. Our experiments show that central and peripheral web regions encode complementary capabilities. Mixture combining both at a ratio of 1:1 achieves 41.4% on average, compared to 39.8% for uniform sampling. Combining structural scores with document-level quality classifier scores further improves performance to 43.8%. These findings demonstrate that web graph topology is a meaningful axis for pretraining data curation, capturing information that is largely orthogonal to existing content-based approaches.
