Search papers, labs, and topics across Lattice.
The paper introduces STING, a framework for diagnosing and augmenting regression test suites in benchmarks like SWE-bench by using semantically altered program variants as diagnostic stressors to identify weaknesses. STING generates new tests that pass on the ground-truth patch but fail on variants that survived the original test suite, ensuring they target under-constrained behaviors without overfitting. Applying STING to SWE-bench Verified reveals that 77% of instances contain surviving variants, and re-evaluating top repair agents with the augmented suites lowers their resolved rates by 4.2%-9.0%, highlighting the importance of test adequacy in benchmark evaluation.
Seemingly impressive results on benchmarks like SWE-bench may be inflated by weak test suites that accept incorrect patches, with a new analysis showing top repair agents' performance dropping by up to 9% when evaluated against more robust tests.
Benchmarks driven by test suites, notably SWE-bench, have become the de facto standard for measuring the effectiveness of automated issue-resolution agents: a generated patch is accepted whenever it passes the accompanying regression tests. In practice, however, insufficiently strong test suites can admit plausible yet semantically incorrect patches, inflating reported success rates. We introduce STING, a framework for targeted test augmentation that uses semantically altered program variants as diagnostic stressors to uncover and repair weaknesses in benchmark regression suites. Variants of the ground-truth patch that still pass the existing tests reveal under-constrained behaviors; these gaps then guide the generation of focused regression tests. A generated test is retained only if it (i) passes on the ground-truth patch, (ii) fails on at least one variant that survived the original suite, and (iii) remains valid under behavior-preserving transformations designed to guard against overfitting. Applied to SWE-bench Verified, STING finds that 77% of instances contain at least one surviving variant. STING produces 1,014 validated tests spanning 211 instances and increases patch-region line and branch coverage by 10.8% and 9.5%, respectively. Re-assessing the top-10 repair agents with the strengthened suites lowers their resolved rates by 4.2%-9.0%, revealing that a substantial share of previously passing patches exploit weaknesses in the benchmark tests rather than faithfully implementing the intended fix. These results underscore that reliable benchmark evaluation depends not only on patch generation, but equally on test adequacy.
