Search papers, labs, and topics across Lattice.
This paper introduces CalibDCD, a calibration framework designed to enhance data contamination detection (DCD) in large language models (LLMs) by addressing the feature shifts that occur during post-training processes. By implementing Multi-View Shift Detection to identify recurring feature shifts and Bounded Feature Correction to mitigate their impact, the authors demonstrate significant improvements in the accuracy of membership predictions. Experimental results indicate that CalibDCD outperforms existing feature-based DCD methods, achieving up to 7.0% increases in AUC and 15.0% in TPR@5%FPR.
CalibDCD reveals that post-training shifts can significantly degrade data contamination detection, but targeted calibration can restore accuracy.
Large language models (LLMs) are trained on massive and largely undisclosed corpora that may contain copyrighted or privacy-sensitive content. Data contamination detection (DCD) therefore aims to determine whether a given text is a member of the pre-training corpus of a target LLM. Recent state-of-the-art DCD methods follow a feature-based paradigm that derives membership features from the input text and the corresponding model output. However, most modern LLMs undergo post-training, such as instruction tuning, preference optimization, and reasoning-oriented training, which can alter model outputs and shift the corresponding membership features, thereby reducing the separability between members and non-members. To address this problem, we propose CalibDCD, a broadly applicable calibration framework for feature-based DCD methods, comprising (1) Multi-View Shift Detection, which identifies recurring feature shifts associated with post-training, and (2) Bounded Feature Correction, which selectively mitigates their influence on membership prediction. Specifically, Multi-View Shift Detection evaluates controlled prompt variants on known non-member texts and consolidates the most informative views to identify recurring feature shifts. Bounded Feature Correction selectively adjusts feature components aligned with the detected shifts and controls the correction extent to preserve useful detection information. Experiments show that CalibDCD consistently improves existing feature-based detectors, with gains of up to 7.0% in AUC and 15.0% in TPR@5%FPR.
