Search papers, labs, and topics across Lattice.
The paper introduces BioAgent Bench, a new benchmark and evaluation suite for assessing AI agents on end-to-end bioinformatics tasks like RNA-seq and variant calling. It uses prompts that specify concrete output artifacts to support automated assessment and stress testing under controlled perturbations such as corrupted inputs and prompt bloat. The authors evaluated both closed-source and open-weight models using an LLM-based grader, finding that while frontier agents can complete multi-step pipelines, they exhibit failure modes under perturbations, and open-weight models may be preferable in privacy-sensitive settings.
Even frontier AI agents struggle with robustness in bioinformatics pipelines, failing under controlled perturbations despite achieving high-level task completion.
This paper introduces BioAgent Bench, a benchmark dataset and an evaluation suite designed for measuring the performance and robustness of AI agents in common bioinformatics tasks. The benchmark contains curated end-to-end tasks (e.g., RNA-seq, variant calling, metagenomics) with prompts that specify concrete output artifacts to support automated assessment, including stress testing under controlled perturbations. We evaluate frontier closed-source and open-weight models across multiple agent harnesses, and use an LLM-based grader to score pipeline progress and outcome validity. We find that frontier agents can complete multi-step bioinformatics pipelines without elaborate custom scaffolding, often producing the requested final artifacts reliably. However, robustness tests reveal failure modes under controlled perturbations (corrupted inputs, decoy files, and prompt bloat), indicating that correct high-level pipeline construction does not guarantee reliable step-level reasoning. Finally, because bioinformatics workflows may involve sensitive patient data, proprietary references, or unpublished IP, closed-source models can be unsuitable under strict privacy constraints; in such settings, open-weight models may be preferable despite lower completion rates. We release the dataset and evaluation suite publicly.
