Search papers, labs, and topics across Lattice.
The paper introduces RESTestBench, a new benchmark for evaluating LLM-generated REST API test cases based on natural language requirements, addressing the limitations of traditional code coverage metrics. It includes three REST services with precise and vague NL requirements and a novel requirements-based mutation testing metric. Experiments using RESTestBench reveal that LLM-generated test effectiveness decreases when interacting with faulty code during refinement, especially for vague requirements.
LLM-generated API tests can be *less* effective when refined against faulty code, especially when requirements are vague, suggesting that blindly incorporating SUT behavior isn't always beneficial.
Existing REST API testing tools are typically evaluated using code coverage and crash-based fault metrics. However, recent LLM-based approaches increasingly generate tests from NL requirements to validate functional behaviour, making traditional metrics weak proxies for whether generated tests validate intended behaviour. To address this gap, we present RESTestBench, a benchmark comprising three REST services paired with manually verified NL requirements in both precise and vague variants, enabling controlled and reproducible evaluation of requirement-based test generation. RESTestBench further introduces a requirements-based mutation testing metric that measures the fault-detection effectiveness of a generated test case with respect to a specific requirement, extending the property-based approach of Bartocci et al. . Using RESTestBench, we evaluate two approaches across multiple state-of-the-art LLMs: (i) non-refinement-based generation, and (ii) refinement-based generation guided by interaction with the running SUT. In the refinement experiments, RESTestBench assesses how exposure to the actual implementation, valid or mutated, affects test effectiveness. Our results show that test effectiveness drops considerably when the generator interacts with faulty or mutated code, especially for vague requirements, sometimes negating the benefit of refinement and indicating that incorporating actual SUT behaviour is unnecessary when requirement detail is high.
