Search papers, labs, and topics across Lattice.
This study critiques the conventional metrics used to report the size of PDF corpora, highlighting a significant discrepancy between document counts and token totals. Analyzing the CC-MAIN-2021-31-PDF-UNTRUNCATED dataset, the authors reveal that a small percentage of documents account for the majority of tokens, with truncation affecting a substantial portion of text. The findings underscore the necessity for dual reporting of corpus statistics to accurately reflect both document and token counts, as current practices can misrepresent the actual text availability in datasets.
A mere 3% of documents contain half of the text in web PDF corpora, revealing a stark imbalance that challenges how we assess corpus size and utility.
PDF corpora advertise their size in tokens but compute every rate they publish (coverage, OCR routing, re-fetch recovery, language mix) per document, and none decomposes its token total. The two units diverge sharply. On CC-MAIN-2021-31-PDF-UNTRUNCATED (7.9M web PDFs, 32.6B tokens), 3.02% of text-bearing documents hold half the tokens (Gini 0.807); documents over 50 pages are 5.00% of the corpus but 53.53% of its text. The PDFs produced by a TeX{} toolchain are 1.66% of documents and 4.05% of the text. The clearest casualty is Common Crawl's truncation cap: it affected 23.06% of documents and 63.08% of the text. Reconstructing the truncated files and extracting both versions, two widely used libraries recover 11.4% and 1.4% of that text; between 72% and 97% of affected documents yield nothing; roughly 55--62% of the corpus's text is lost. Under the 5 MiB cap adopted in March 2025, 30.19% of tokens would still be truncated, and recovery on those documents rises only from 3.3% to 13.2%. We recommend that corpus statistics be reported in both units: documents and tokens.
