Search papers, labs, and topics across Lattice.
This paper investigates the feasibility of poisoning pretraining data for language models (LMs) using public discussion interfaces, highlighting a gap in previous research that primarily focused on established data sources like Wikipedia. By introducing HalfLife, a novel analysis method, the authors measure the inclusion of adversarial content in web-crawled training data, revealing that malicious content can be effectively injected at scale. The findings underscore the critical need for robust data curation processes to mitigate the risks associated with such poisoning attacks on LMs.
Poisoning pretraining data through public discussion interfaces poses a significant threat, with the potential for undetectable harmful behaviors in language models.
Poisoning pretraining data can introduce harmful behaviors to LMs that are difficult to detect and mitigate. Prior work on poisoning pretraining data has largely exploited established data sources such as Wikipedia, which do not represent the large scale and heterogeneity typical of pretraining corpora, and has ignored the interaction between poisoned data and data curation pipelines. We demonstrate that poisoning attacks on pretraining data are feasible beyond this limited setting through an existing web-scale content injection mechanism: public discussion interfaces. Additionally, to measure whether malicious content is included after web crawling and data curation, we introduce HalfLife, a novel analysis for estimating adversarial content inclusion in web-crawl based LM training data. We use HalfLife to explore the feasibility of poisoning pretraining corpora at web scale through open discussion interfaces. Our analysis demonstrates the importance of estimating whether poison injections are included in pretraining data, and establishes third-party webpage content as a possible vector for attacking language model pretraining.
AI2
UW