Search papers, labs, and topics across Lattice.
This paper introduces a benchmark, encompassing the HC3 and ELI5 datasets, to evaluate the performance of various AI-generated text detection methods, including classical classifiers, fine-tuned transformers, XGBoost, perplexity-based methods, and LLM-as-detector prompting. The study reveals that while transformer models excel in-distribution, their performance significantly drops with domain shifts, and that LLM-based detectors suffer from generator-detector identity bias. A key finding is the lack of robust generalization across domains and LLM sources for all tested methods, highlighting the need for more adaptable detection techniques.
AI-generated text detectors that seem perfect in the lab fall apart in the real world, with no single method generalizing across domains or even different LLMs.
The rapid proliferation of large language models (LLMs) has created an urgent need for robust and generalizable detectors of machine-generated text. Existing benchmarks typically evaluate a single detector on a single dataset under ideal conditions, leaving open questions about cross-domain transfer, cross-LLM generalization, and adversarial robustness. We present a comprehensive benchmark evaluating diverse detection approaches across two corpora: HC3 (23,363 human-ChatGPT pairs) and ELI5 (15,000 human-Mistral-7B pairs). Methods include classical classifiers, fine-tuned transformer encoders (BERT, RoBERTa, ELECTRA, DistilBERT, DeBERTa-v3), a CNN, an XGBoost stylometric model, perplexity-based detectors, and LLM-as-detector prompting. Results show that transformer models achieve near-perfect in-distribution performance but degrade under domain shift. The XGBoost stylometric model matches performance while remaining interpretable. LLM-based detectors underperform and are affected by generator-detector identity bias. Perplexity-based methods exhibit polarity inversion, with modern LLM outputs showing lower perplexity than human text, but remain effective when corrected. No method generalizes robustly across domains and LLM sources.
