Search papers, labs, and topics across Lattice.
BioSecBench-Surveillance introduces a comprehensive benchmark consisting of 100 evaluations designed to assess AI agents' ability to accurately infer analysis pipelines from raw genomic sequencing data in the context of pathogen surveillance. The evaluations cover a range of tasks, including taxonomic classification and genetic-engineering detection, with results indicating that even the best-performing models, such as Opus 4.8 and GPT-5.5, achieved only about 50% accuracy. This benchmark not only highlights the current limitations of AI in genomic analysis but also establishes a verifiable standard for future developments in AI-assisted pathogen surveillance.
AI agents struggle to surpass 50% accuracy in genomic surveillance tasks, revealing significant gaps in their analytical capabilities.
As pathogen genomic surveillance scales, the bottleneck is shifting from data generation to analysis. We present BioSecBench-Surveillance, a verifiable benchmark of 100 evaluations testing whether AI agents can infer the right analysis pipeline from raw sequencing data and surveillance context. Each evaluation gives an agent only the data and context a human analyst would have, then grades its structured answer deterministically. The tasks span seven categories, from taxonomic classification to genetic-engineering detection, across diverse sample types and sequencing technologies. Across 3,962 gradable attempts from sixteen model-harness pairs, the strongest configuration cleared only about half. Opus 4.8 with PI led at 50.2 percent, with a 95 percent confidence interval of 40.1 to 60.3 percent across 83 evaluations, tied with GPT-5.5 with Codex at 50.2 percent, with a 95 percent confidence interval of 40.8 to 59.6 percent, followed by Opus 4.7 with PI at 49.6 percent, with a 95 percent confidence interval of 40.0 to 59.2 percent, and Sonnet 4.6 with PI at 48.6 percent, with a 95 percent confidence interval of 38.9 to 58.3 percent. Even when agents invoked the correct workflows, their mistakes came from the choices around them, such as which references, thresholds, filters, and normalization to apply. BioSecBench-Surveillance provides a standard for measuring whether agents can be trusted to perform genomic surveillance when the next outbreak arrives.
