Search papers, labs, and topics across Lattice.
This paper introduces QTestBench, a novel framework for evaluating the quality of LLM-generated quantum test suites, addressing the unique challenges of quantum software testing such as probabilistic outcomes and numerical tolerances. By treating independent benchmark checkers as ground truth and injecting candidate solutions at runtime, QTestBench assesses the discriminative power of generated tests across syntactic, semantic, and behavioral dimensions. Results indicate that while LLM-generated quantum tests are largely executable, their ability to distinguish correct from faulty implementations significantly lags, highlighting a critical gap in current LLM capabilities for quantum software verification.
LLM-generated quantum tests are executable but fail to reliably distinguish correct from faulty code, with the best models achieving only 71% and 34% test quality on separate benchmarks.
Large language models (LLMs) have shown strong results in classical code generation and are increasingly used to write the unit tests that safeguard that code. Whether they can do the same for quantum software is less clear, because quantum programs are probabilistic, depend on numerical tolerances, and are judged by domain-specific checkers, so a well-formed test can still disagree with the benchmark's verdict. Existing quantum benchmarks measure whether a model produces a correct solution, but not whether its tests can distinguish correct implementations from faulty ones. We introduce QTestBench, a framework that evaluates LLM-generated quantum test suites against independent benchmark checkers, never treating the tests themselves as ground truth. For each task, QTestBench prompts models to produce syntactic, semantic, and behavioral tests, injects candidate solutions at runtime, and compares each test verdict with the checker's label under three validation cohorts, each answering a distinct diagnostic question. We benchmark five LLMs on eight PennyLane QHack challenges and ten Qiskit HumanEval tasks. Generated suites almost always execute, but discrimination lags behind: the best generator reaches a test quality of 0.71 on QHack and 0.34 on Qiskit HumanEval, so executability alone overstates test quality. Oracle-verified synthetic mutants stabilize bug-detection estimates when natural faulty solutions are scarce, as on three of ten Qiskit tasks that yield none. QTestBench offers a reproducible, cross-framework benchmark for the trustworthiness of LLM-generated quantum tests.
