Search papers, labs, and topics across Lattice.
This paper investigates Contamination Detection via output Distribution (CDD) for small language models (70M-410M) to identify data contamination by measuring the peakedness of sampled outputs. Through controlled contamination experiments on GSM8K, HumanEval, and MATH, the study reveals that CDD's effectiveness hinges on whether fine-tuning leads to verbatim memorization. Specifically, low-rank adaptation allows models to learn from contaminated data without memorization, rendering CDD ineffective, while sufficient fine-tuning capacity inducing memorization enables strong detection accuracy.
Parameter-efficient fine-tuning can create undetectable data contamination in small language models, even when the data is verifiably contaminated.
CDD, or Contamination Detection via output Distribution, identifies data contamination by measuring the peakedness of a model's sampled outputs. We study the conditions under which this approach succeeds and fails on small language models ranging from 70M to 410M parameters. Using controlled contamination experiments on GSM8K, HumanEval, and MATH, we find that CDD's effectiveness depends critically on whether fine-tuning produces verbatim memorization. With low-rank adaptation, models can learn from contaminated data without memorizing it, and CDD performs at chance level even when the data is verifiably contaminated. Only when fine-tuning capacity is sufficient to induce memorization does CDD recover strong detection accuracy. Our results characterize a memorization threshold that governs detectability and highlight a practical consideration: parameter-efficient fine-tuning can produce contamination that output-distribution methods do not detect. Our code is available at https://github.com/Sela-Omer/Contamination-Detection-Small-LM
