Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces "deployment-complete benchmarking," a framework to assess whether benchmark results reliably inform deployment decisions. It argues that standard benchmarks often fail to guarantee consistent actions across different evidence fibers, leading to poor transferability to real-world deployment scenarios. Through audits of public datasets like Tox21 and Matbench, the authors demonstrate widespread incompleteness and propose a "certify-then-acquire" strategy to reduce false decisions in deployment.
Standard benchmarks are often misleading because even zero benchmark error can certify less than half of candidates in real-world deployments.
Benchmarks increasingly guide deployment, procurement and scientific screening, yet a score supports only the response it records, not necessarily the deployment action. We introduce deployment-complete benchmarking, which tests whether benchmark evidence determines a deployment action. A benchmark is complete for a claim exactly when the action is constant on each evidence fiber; mixed fibers expose missing deployment information, and completion curves quantify the evidence required to resolve ambiguity. In controlled response spaces, benchmark-channel conformal coverage of 94.98% transferred poorly to an unmeasured deployment channel (10.07%), whereas response-rank intervals achieved 94.91% coverage; even zero benchmark error certified only 45.4% of candidates at the largest residual size. Public audits revealed incompleteness, including 97.9% mixed Tox21 fibers and zero median certifiable fraction in main Matbench and JARVIS audits. In held-out replays, certify-then-acquire reduced false decisions from 1.19% to 0.027% in Tox21 and from 20.3% to 0.128% in JARVIS, while changing model choice and identifying deployment-relevant probes. Deployment-ready benchmarks should report evidence, supported actions, ambiguity and completion cost rather than scores alone.