Search papers, labs, and topics across Lattice.
This paper evaluates a novel approach called Spec-Driven Test Generation, which enhances LLM-based agents' ability to generate high-quality tests by requiring them to first reason about and document the code's pre-conditions, post-conditions, and undefined behaviors. The method significantly improves bug detection rates by 9.8 percentage points and branch coverage by 2.5 percentage points compared to traditional test generation agents, as evidenced by empirical evaluations on production bugs from Google. Furthermore, the generated test suites outperform both baseline and human-authored tests in a majority of cases, highlighting the effectiveness of incorporating structured reasoning into the test generation process.
Spec-Driven Test Generation boosts bug detection rates by nearly 10% by making LLMs reason about code contracts before generating tests.
LLM-based agents are increasingly used for coding tasks, where they have outperformed many classical approaches and scaled to repository-level tasks, such as test generation. However, when directly prompted to generate tests, these agents can fail to reason about the code and its underlying contracts, thereby missing edge cases and behavioral boundaries that affect test quality. To address this limitation, we propose Spec-Driven Test Generation, where we instruct an agent to first reason about -- and explicitly document -- code pre-conditions, post-conditions, and undefined behaviors. This intermediate semi-formal specification acts as a cognitive scaffold to guide subsequent test generation. Our evaluation on production bugs from Google shows that the spec-driven agent can deliver a 9.8 percentage points ($p = 0.0352$) improvement in bug detection rate and a 2.5 percentage point ($p = 0.0034$) improvement in branch coverage, compared to a traditional test generation agent baseline. Using LLM-as-a-Judge, we further show that test suites generated by the spec-driven agent are superior to the baseline and human-authored tests in 77.8% and 56.7% of the cases, respectively, and demonstrated improvements on following best practices, readability, and edge-case coverage.
