Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
We introduce TestPrism, a novel evaluation framework for LLM-generated code tests that moves beyond single reference solutions to account for implementation diversity. By employing a Joint Success Function requiring tests to fail initial states, accept all valid candidates, and reject all invalid ones, we reveal significant overestimation of test quality by current methods. Our proposed TestHelix framework, utilizing heterogeneous synthesis and recursive self-improvement, demonstrably improves evaluation accuracy by 8.67-9.00 percentage points.
LLM test generation quality is overestimated by 30% when evaluated against single references, with a new metric revealing widespread missed behaviors and faulty assertions.
Large language model (LLM) coding agents have advanced test generation across diverse programming tasks. However, the common practice of evaluating tests against a single reference solution overlooks alternative valid implementations and can overstate test quality. We introduce TestPrism, comprising 300 test tasks from 17 sources and 3000 candidate implementations, evenly split between valid and invalid solutions. Its primary metric, Joint Success Function, requires the generated tests to fail on the initial program state, accept every valid candidate, and reject every invalid candidate. Across fourteen baseline coding agent configurations, Joint Success Function reaches only 28.00%, whereas single reference success reaches 59.67%. Our analysis reveals missed behaviors, unsupported assertions, and faulty test construction. To address these weaknesses, we introduce TestHelix, which combines heterogeneous synthesis of test and repair pairs with peer cross validation and recursive self improvement (RSI). Across two models, TestHelix improves Joint Success Function by 8.67 to 9.00 percentage points over the native harness comparators in the TestHelix evaluation