Search papers, labs, and topics across Lattice.
This study audits the reliability of execution feedback in feedback-driven test generation for large language models (LLMs), revealing that using a single accepted program as ground truth can lead to misleading fault detections and inflated evolutionary gains. By evaluating 142 development tasks and comparing various feedback mechanisms, the authors find that generated outputs align with expert panels only 27.79% to 50.12% of the time, and that independent resampling methods significantly outperform mutation-based evolution. The research highlights the importance of robust evaluation protocols, proposing an audit-and-placebo method to better isolate genuine improvements from artifacts of the evaluation process.
Relying on a single oracle for feedback can inflate perceived gains in LLM test generation by nearly 15 percentage points, masking the true effectiveness of evolution strategies.
Execution feedback is often treated as a self-verifying signal for improving LLM-generated tests. However, when generated inputs are executed on a single accepted program and its outputs are used as ground truth, invalid or underspecified inputs can create spurious fault detections and apparent evolutionary gains. We audit this failure mode in feedback-driven test generation using 142 development tasks, 114 locked external tasks, and 138 held-out tasks, with two code models, three seeds, and fault-cross-fitted real submissions. On external inputs for which three accepted implementations agree, generated outputs match the panel on only 27.79% and 50.12% of cases. A single-reference oracle inflates the measured gain from evolution by 9.46-14.85 percentage points; after auditing, equal-budget independent resampling outperforms mutation-based evolution by 6.01-18.83 points. We further compare a genuine three-round feedback loop with a density-matched placebo. External Real-Placebo differences are +0.13 and -0.50 points, while held-out differences are +1.99 and +0.28 points and do not provide robust evidence of fine-grained feedback benefit. A blinded semantic audit by two software engineering doctoral students classifies 94.41% of panel-disconfirmed inputs as invalid but 3.60% as valid, showing that panel disagreement is informative but not semantic proof. We propose an audit-and-placebo protocol that separates verifier artifacts, interaction scaffolding, and grounded feedback credit in evaluations of self-evolving test generators.
