Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
QC-Stark evaluates 10 leading LLMs across 11 fully auto-verifiable quantum computing tasks—including circuit synthesis, compilation, and error correction—across 2,750 execution-backed trials. The evaluation reveals severe capability dissociations, showing that aggregate leaderboard rankings fail to predict task-specific performance, with Spearman rank correlations proving statistically insignificant across 4 of the 11 tasks. Validated via 2-parameter Item Response Theory and prompt sensitivity analyses, the findings expose how monolithic performance metrics obscure critical domain-specific failures in specialized technical domains.
Aggregate benchmark scores actively mislead in quantum computing, failing to predict individual model competence across more than a third of tasks like circuit compilation and quantum error correction.
We introduce QC-Stark, a benchmark for evaluating large language models (LLMs) on 11 quantum computing (QC) tasks, spanning circuit construction, debugging, compilation, error correction, and simulation. Across 2,750 evaluations (10 models $\times$ 11 tasks x 5 difficulty levels x 5 seeds), we find that overall rankings mask substantial per-task variation. The Spearman correlation between overall and per-task rankings is statistically insignificant for 4 out of the 11 tasks included in this benchmark. A 2-parameter Item Response Theory (IRT) model validates measurement quality, and prompt sensitivity analysis confirms ranking robustness across prompt conditions. All tasks are auto-verifiable via execution, thus not requiring any manual evaluation. We make the code and data publicly available on Huggingface.