Search papers, labs, and topics across Lattice.
This paper outlines the first Poisoned Chalice of LLM Evaluation Competition, which addresses the critical issue of data contamination in evaluating large language models (LLMs) for software engineering tasks. By framing contamination detection as a white-box membership inference task, the competition provides participants with curated datasets and baseline attacks to develop robust methods for identifying prior exposure to training data. The initiative aims to foster community engagement and improve the reliability of LLM evaluations in software engineering, ultimately leading to more trustworthy assessments of model capabilities.
The competition reveals that many LLM evaluations may be fundamentally flawed due to undetected data contamination, challenging the validity of current benchmarks.
Large language models are increasingly used to evaluate and support software engineering tasks, yet the validity of these evaluations is often undermined by uncertainty about whether benchmark instances were seen during pretraining. This can lead to data contamination, which may inflate performance and result in misleading conclusions about model capability. Despite this, the training corpora of many modern models are only partially disclosed, making direct decontamination infeasible. This creates a need for practical methods that can detect a large language models'prior exposure to training data without access to the full training corpus. To address this challenge, we organize the first Poisoned Chalice of LLM Evaluation Competition, co-located with the FSE-AIWare 2026 Competition Track. The competition frames contamination detection as a white-box membership inference task on source code and provides participants with curated datasets, target models, baseline attacks, and a final evaluation on a held-out model and dataset. This design encourages methods that generalize beyond superficial dataset artifacts and beyond a single training setting. This paper reports the setup and results of the competition. More broadly, the competition aims to catalyze the community around trustworthy LLM evaluation for software engineering.
