Search papers, labs, and topics across Lattice.
This paper introduces SrDetection, a self-referential framework designed to detect data leakage in code large language models (Code LLMs) by contrasting model behavior on original benchmark samples with semantically equivalent variants. Unlike existing methods that rely on proprietary data or brittle heuristics, SrDetection operates in both gray-box and black-box settings, providing a more robust and generalizable approach to leakage detection. The framework significantly improves detection performance, achieving an average F1 score increase of 21.52 points in gray-box scenarios and 14.46 points in black-box scenarios compared to strong baselines, while revealing benchmark-specific leakage patterns in widely used Code LLMs.
SrDetection uncovers hidden data leakage patterns in Code LLMs, boosting detection accuracy by over 21 points without relying on fragile heuristics.
Evaluating code large language models (Code LLMs) requires reliable detection of data leakage, where benchmark performance is artificially inflated by exposure to benchmark data during pre-training. Existing approaches either assume access to proprietary training corpora, rely on brittle heuristics such as timestamp filtering, or use external reference sets with manually tuned, non-generalizable thresholds. To address these limitations, we introduce \textbf{SrDetection}, a unified \textbf{s}elf-\textbf{r}eferential leakage detection framework for both gray-box (access to model logits) and black-box (access to model outputs) settings. SrDetection generates semantically equivalent variants of a benchmark sample and detects leakage by contrasting the model's behavior on the original versus its variants, flagging cases where the original is disproportionately easier for the model. We further design a controlled leakage detection testbed and evaluate SrDetection in this environment. Across different models and training stages, SrDetection improves average F1 by 21.52 points in the gray-box setting and 14.46 points in the black-box setting over strong baselines, demonstrating robust, threshold-independent leakage detection. Finally, a gray-box study of 15 widely used Code LLMs on four popular benchmarks reveals benchmark-specific leakage patterns beyond prior overlap-based analyses\footnote{\footnotesize Source code and data are available at https://github.com/SMinL/SrDetectionCode
