Search papers, labs, and topics across Lattice.
This study empirically compares the fault-detection capabilities of LLM-generated unit tests against human-written tests across three Python benchmarks, revealing that LLMs with retrieval-augmented context detect faults in 69% of cases, significantly outperforming human tests at 17.2%. The research highlights that traditional coverage metrics are inadequate indicators of test quality, as both LLMs and humans achieve similar coverage rates. By analyzing the conditions under which each method excels, the authors underscore the importance of retrieval context and robust benchmark design in evaluating test effectiveness.
LLM-generated unit tests can detect faults nearly four times more effectively than human-written tests, challenging the assumption that coverage is a reliable measure of test quality.
Large language models (LLMs) have shown considerable promise for automated unit test generation, yet their practical effectiveness relative to human-written tests remains poorly understood. Existing evaluations commonly rely on coverage-oriented benchmarks that do not assess fault-detection capability directly. We present an empirical comparison of LLM-generated and human-written unit tests across three complementary Python benchmarks: 29 real historical bugs from BugsInPy, a function-level benchmark drawn from python-slugify and packaging, and a controlled paired benchmark. Our generation pipeline couples Gemini 2.5 Flash with a lightweight lexical retrieval mechanism that supplies bug-relevant context at generation time. Across eight quality dimensions, LLM-generated tests with retrieval-augmented context detect faults in 69% of cases compared to 17.2% for general-purpose human-written tests (Fisher's exact, $p < 0.001$, Cohen's $h = 1.10$). Critically, line and branch coverage are nearly identical between the two approaches (84.8% vs. 88.5% and 75.2% vs. 82.1%), confirming that coverage is an insufficient proxy for fault-detection capability. We discuss the conditions under which each approach excels, characterize their complementary strengths, and identify the critical role of retrieval context and reproducible benchmark construction in meaningful test-quality evaluation.
