Search papers, labs, and topics across Lattice.
The paper investigates the common practice of using a single, fixed extractor for converting HTML to text in LLM pretraining datasets, finding it suboptimal in terms of coverage and utilization of web data. They demonstrate that different extractors, while yielding similar performance on standard language understanding tasks, result in substantially different sets of surviving pages after filtering. By taking a union of extractions from multiple extractors, the authors achieve a 71% increase in token yield for DCLM-Baseline while maintaining benchmark performance, and also show significant downstream task performance improvements by choosing appropriate extractors for structured content.
Sticking to a single HTML-to-text extractor in your LLM pretraining pipeline could be leaving 71% of the data on the table.
One of the first pre-processing steps for constructing web-scale LLM pretraining datasets involves extracting text from HTML. Despite the immense diversity of web content, existing open-source datasets predominantly apply a single fixed extractor to all webpages. In this work, we investigate whether this practice leads to suboptimal coverage and utilization of Internet data. We first show that while different extractors may lead to similar model performance on standard language understanding tasks, the pages surviving a fixed filtering pipeline can differ substantially. This suggests a simple intervention: by taking a Union over different extractors, we can increase the token yield of DCLM-Baseline by up to 71% while maintaining benchmark performance. We further show that for structured content such as tables and code blocks, extractor choice can significantly impact downstream task performance, with differences of up to 10 percentage points (p.p.) on WikiTQ and 3 p.p. on HumanEval.
Apple ML
Google Research
Stanford HAI