Search papers, labs, and topics across Lattice.
This paper introduces a scalable framework for subdocument deduplication in large language model pretraining, addressing the limitations of existing methods that struggle with cross-shard duplicates and fixed retention policies. By employing natural-boundary segmentation and normalized exact hashing, the framework identifies duplicate groups and implements a frequency- and length-aware retention strategy that adapts to varying repetition patterns. Experiments demonstrate that models trained on data processed with this method outperform those trained on traditional deduplication approaches, highlighting the critical role of tailored copy-retention strategies in enhancing model performance.
Retaining more copies of low-frequency content while aggressively pruning high-frequency duplicates can significantly boost model performance during pretraining.
Large-scale pretraining corpora contain substantial duplicate content. Although document-level deduplication is widely used, removing subdocument-level redundancy remains challenging. At corpus scale, suffix-array-based methods are commonly applied independently within shards, leaving cross-shard duplicates undetected and making the resulting retention behavior sensitive to the sharding configuration. Hash-based methods enable global exact duplicate counting, but often rely on fixed copy-retention policies that cannot accommodate heterogeneous repetition patterns. We propose a scalable subdocument deduplication framework that decouples duplicate detection from copy retention. It identifies duplicate groups through natural-boundary segmentation, normalized exact hashing, and distributed aggregation, and then applies an explicit frequency- and length-aware retention policy that allocates an adaptive copy budget to each group, retaining more copies of low-frequency or short repetitions while more aggressively deleting high-frequency or long ones. Experiments on FineWeb-Edu and a code-containing web corpus show that models trained on data processed by our method achieve the best overall performance among the evaluated settings. These results underscore the importance of explicit copy-retention control.
