Search papers, labs, and topics across Lattice.
This paper critiques the existing methods for evaluating contamination mitigation in benchmark models, arguing that the G-AP metric fails to accurately capture per-question performance due to its averaging approach. The authors introduce SA-PPG, a new metric that stratifies solve probabilities by sampling and aggregates them based on the clean model's performance, offering a more nuanced understanding of contamination effects. Their findings reveal that previous restoration strategies significantly overestimate model capabilities, while their proposed RailCap method achieves superior performance by dynamically capping token probabilities during generation to enhance distribution diversity.
Previous contamination mitigation strategies may inflate model performance by over 40%, but a new evaluation method reveals the true extent of this overestimation.
Test data from public benchmarks inevitably leaks into pretraining corpora, inflating evaluation scores once memorized. Contamination mitigation evaluation intervenes in the decoding process to suppress memorization and restore a contaminated model's genuine capability, but its prevailing metric, the G-AP (Gap of Aggregate Performance), is flawed. Discrete correct/incorrect readouts cannot characterize per-question performance, averaging before differencing lets over- and under-suppression cancel out, and uniform per-question weighting invites strategies to push solve probabilities onto the clean model's high-frequency values. We propose SA-PPG (Stratified Aggregate of Per-question Probability Gaps): estimate each question's solve probability by sampling, difference it against the clean model per question, and aggregate within groups defined by the clean model's solve probability. Existing mitigation strategies first estimate where contamination lies and then operate on the estimate, so they are only as correct as the estimate. RailCap instead judges contamination during generation: whenever a sample falls back onto the greedy trajectory, the next trajectory token is capped to the runner-up, accumulating suppression until the response distribution becomes sufficiently dispersed. Across multiple contaminated models and benchmarks, SA-PPG reveals that prior strategies' restoration is substantially overestimated, while RailCap attains the lowest SA-PPG.
