Search papers, labs, and topics across Lattice.
This paper introduces a perturbation-based method to quantify the "memorization advantage" of code LLMs, measuring the performance difference between likely seen and unseen inputs. They evaluate 8 open-source code LLMs on 19 benchmarks, finding significant variations in sensitivity to memorization across models and tasks. Surprisingly, the CVEFixes and Defects4J benchmarks show low memorization advantage, suggesting models generalize rather than simply memorize those datasets.
Code LLMs don't just memorize training data – some generalize far better than others, and even "leaky" datasets like CVEFixes show surprisingly low memorization advantage.
The lack of transparency about code datasets used to train large language models (LLMs) makes it difficult to detect, evaluate, and mitigate data leakage. We present a perturbation-based method to quantify memorization advantage in code LLMs, defined as the performance gap between likely seen and unseen inputs. We evaluate 8 open-source code LLMs on 19 benchmarks across four task families: code generation, code understanding, vulnerability detection, and bug fixing. Sensitivity patterns vary widely across models and tasks. For example, StarCoder reaches high sensitivity on some benchmarks (up to 0.8), while QwenCoder remains lower (mostly below 0.4), suggesting differences in generalization behavior. Task categories also differ: code summarization tends to show low sensitivity, whereas test generation is substantially higher. We then analyze two widely discussed benchmarks, CVEFixes and Defects4J, often suspected of leakage. Contrary to common concerns, both show low memorization advantage across models: CVEFixes remains below 0.1, and Defects4J is lower than other program repair benchmarks. These results suggest that, for these datasets, models may rely more on learned generalization than direct memorization. Overall, our findings provide evidence that memorization risk is highly task- and model-dependent, and highlight the need for stronger evaluation protocols, especially in security-focused settings.
