Search papers, labs, and topics across Lattice.
This paper introduces OpenBioRQ, a novel benchmark comprising 12,553 unsolved biomedical research questions designed to evaluate the faithfulness and abstention of agentic models in a retrieval-grounded setting. The benchmark uniquely focuses on open questions without fixed answer keys, revealing that while models can resolve citations, they often link to incorrect papers, with a significant 15.9% error rate. Results show that even advanced models struggle with this challenging dataset, achieving only 17% success on the hardest questions, highlighting a critical gap in current evaluation methods for agentic AI systems.
Agentic models may resolve citations, but they still mislink to the wrong papers 15.9% of the time, exposing a critical flaw in current AI evaluation benchmarks.
A working citation looks like proof -- but the fact that a link resolves does not mean the cited paper supports the claim. I find that current agentic models rarely fabricate citations (over $99\%$ resolve), yet roughly $15.9\%$ link to the wrong paper. Existing benchmarks miss this failure mode: when a question has a fixed answer key, a model can reproduce the expected source from that key rather than independently verifying that the source supports the claim. I introduce \textbf{\openbiorq{}}, a retrieval-grounded agentic benchmark of $12{,}553$ unsolved biomedical research questions across $12$ domains that treats open questions as a faithfulness-and-abstention probe. To my knowledge, this is the first biomedical benchmark to combine an agentic setting -- where the model must issue multiple tool calls -- with unsolved questions that have no answer key. Openness is verified against real follow-up evidence rather than a model's parametric knowledge. Difficulty is empirical: I anchor it on questions that three open-weight reference models fail to answer, rather than on subjective hardness labels. On this hardest subset, held-out models from the same lineage as the difficulty anchors solve only ~17%, while three independent frontier agents (Gemini-3-Pro, Opus-4.7, GPT-5.5) span a wide 29-60% range. The benchmark is thus hard, non-saturating (the best agent still leaves ~33-40\% unsolved), and discriminating across capability tiers. Beyond difficulty, I observe agentic collapse on the hardest questions, where agents stop using their tools. For the most collapse-prone model, blocking tool access entirely barely changes its score -- so tools stop paying off exactly where they are needed most. A frozen per-question checklist raises inter-judge agreement from Spearman 0.35 to 0.82.
