Search papers, labs, and topics across Lattice.
This paper introduces Qolumbina, a benchmark infrastructure designed for quantum software testing (QST) that curates and refines 40 scalable quantum programs from open-source repositories. By establishing systematic selection and testing protocols, Qolumbina enables rigorous empirical evaluations that reflect contemporary software development practices, addressing the limitations of existing benchmarks. The empirical study reveals that Qolumbina effectively captures diverse testing-relevant properties and supports scalability analysis, demonstrating its utility in execution-cost and fault-detection studies across different QST approaches.
Qolumbina reveals that existing quantum software testing benchmarks are inadequate, paving the way for more robust evaluations of scalable quantum programs.
Quantum software testing (QST) checks whether quantum programs behave according to their intended specifications. A key requirement for QST research is a benchmark that supports rigorous empirical evaluation on programs that are testable and better reflect current software development practices. However, existing studies heavily rely on small hard-coded or circuit-level benchmarks, while available quantum programs are scattered across repositories without clear selection criteria, which limits fair comparison and systematic reproducibility. To this end, we present Qolumbina, a benchmark infrastructure for controlled QST experiments on scalable quantum programs. Qolumbina curates 40 programs from open-source repositories, turns them into test-ready subjects through systematic selection, refactoring, specifications, test case examples, unit tests, and standardized interfaces. We also propose QST-oriented criteria to characterize quantum programs along functionality, output behavior, development complexity, and quantum-specific execution complexity. Using these criteria, our empirical study shows that Qolumbina covers diverse testing-relevant properties and supports scalability analysis beyond fixed-size circuit benchmarks. Through controlled experiments with two recent QST approaches, we demonstrate the feasibility of using Qolumbina for execution-cost and fault-detection studies, and highlight backend-dependent effects that can influence QST result interpretation.
