Search papers, labs, and topics across Lattice.
This paper investigates the unique token repetition patterns in LLM-generated text, hypothesizing that a strong aversion to repetition serves as a distinguishing feature from human writing. By introducing the Telescope Perplexity metric, the authors demonstrate that this signature emerges early in pre-training and can effectively differentiate between LLM and human text in a zero-shot context. The results indicate that Telescope Perplexity achieves state-of-the-art performance across various datasets, outperforming existing detection methods in both accuracy and efficiency.
LLM-generated text reveals a unique token repetition signature that can be leveraged for highly effective zero-shot detection, outperforming traditional methods.
Distinguishing Large Language Model (LLM) generated text from human writing is a critical and difficult challenge. While LLMs are trained to write like humans, we hypothesize that this training leaves an indelible mark. LLMs develop a particularly strong aversion to token repetition very early in training. This bias persists as a ''Vestigial Heuristic'' (a developmental artifact) that is activated in LLM-generated text, separating LLM from human writing. To probe this phenomenon, we introduce Telescope Perplexity, a metric that evaluates the token repetition of the model, $P(s_i | s_{1:i})$ . Our empirical investigation reveals that the Telescope Perplexity signature emerges early in pre-training, and Telescope Perplexity empirically enables highly effective zero-shot LLM detection. We show state-of-the-art or competitive performance across diverse datasets (including modern evaluation sets we introduce), reference models, and perturbation schemes with greater efficiency than other methods.
