Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for evaluating decontamination methods in large language models (LLMs) that focuses on sample-level performance rather than aggregate accuracy, revealing significant variations in model behavior. The authors present Uncertainty-Based Decontamination (UBD), which utilizes deep ensembles to estimate per-sample memorization without needing an uncontaminated model, effectively addressing the issue of data contamination in LLM evaluations. Experiments show that UBD significantly improves the alignment of output distributions with those of uncontaminated models, outperforming existing baselines while maintaining performance on clean data.
UBD reveals that traditional accuracy metrics can mask critical sample-level discrepancies in model behavior, leading to a more nuanced understanding of data contamination effects.
Benchmark-based evaluation is the dominant paradigm for assessing large language model (LLM) capabilities, yet data contamination inflates reported performance and undermines fair comparison. Existing decontamination methods are evaluated solely through aggregate accuracy, which can obscure substantial differences in per-sample model behaviour, and many require access to an uncontaminated model. In this paper, we propose a sample-level evaluation framework for decontamination that complements accuracy-based assessment with distributional distance metrics, measuring how closely a decontaminated model recovers the output distribution of an uncontaminated model on each sample. Building on this framework, we introduce Uncertainty-Based Decontamination (UBD), a family of methods that leverage deep ensembles of the contaminated model to estimate per-sample memorization without requiring a uncontaminated model or knowledge of which samples are contaminated. UBD estimates a per-sample correction scalar from ensemble uncertainty, which is used to construct a debiased target distribution that suppresses the inflated probability mass on correct answers induced by contamination. This target is then used either as a post-hoc output correction (debiasing) or as a soft training signal for parameter update (unlearning). Experiments on MMLU-Pro and MATH-MCQA across multiple LLM backbones demonstrate that UBD produces per-sample output distributions substantially closer to those of an uncontaminated model than paraphrasing or choice-permutation baselines, while preserving model performance on uncontaminated data.
