Search papers, labs, and topics across Lattice.
This paper introduces Sampled-BPE, a novel auditing pipeline designed to efficiently analyze Chinese web-scale corpora for token-level pollution, addressing the challenges posed by the vast size and rapid changes in web content. By sampling a small subset of data and training a Byte Pair Encoding (BPE) tokenizer, the method achieves a remarkable 148.4脳 speedup and 35.8脳 memory reduction while maintaining a low relative error of 4.25% in pollution estimates. The application of Sampled-BPE across multiple Chinese corpora reveals significant and uneven levels of pollution, underscoring the need for ongoing monitoring in the rapidly evolving landscape of Chinese web content.
Auditing Chinese web content reveals pervasive pollution that shifts over time, challenging the integrity of LLM training data.
Chinese web pollution has surfaced in LLMs, motivating audits of upstream Chinese corpora. However, auditing such corpora faces three challenges: (1) their web-scale size makes full scan costly; (2) prior analyses are often too coarse to expose token-level pollution; (3) Chinese web pollution is implicit and rapidly changing. We propose Sampled-BPE, a lightweight token-level auditing pipeline that sample a small subset and train BPE tokenizer to surface polluted tokens. Experiments show that Sampled-BPE preserves usable estimates while substantially reducing runtime and memory: a 148.4 $\times$ speedup and a 35.8 $\times$ memory reduction induce only 4.25% relative error for pollution categories. We apply the pipeline to 11 open Chinese corpora and 6 Chinese Common Crawl snapshots from 2021 to 2026. The audit reveals widespread but uneven pollution across open corpora, as well as highly polluted and temporally shifting Chinese web content. We further release a hierarchical Chinese web token dataset with 660k+ token records, each with web context, category, and explanation fields, organized as trees to support review and tracing of pollution.
Tsinghua AI