Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study evaluates Claude Fable 5's performance on eight biomedical benchmarks, revealing significant refusal rates that vary from 8.0% to 99.4% depending on the task. By employing deterministic scoring against fixed answer keys, the authors demonstrate that once refusals are excluded, Fable 5 outperforms its predecessors and GPT-5 across all metrics. The findings highlight that Fable 5's limitations in biomedical applications stem more from its willingness to engage with questions rather than its inherent capabilities when it does respond.
Claude Fable 5's refusal rates can skew performance metrics, revealing that its true accuracy is masked by its reluctance to answer certain biomedical questions.
Frontier language models are increasingly evaluated on biomedical benchmarks, but two problems undermine most published evaluations: legacy benchmarks are near-saturated, and open-ended responses are graded by other language models. We evaluate Claude Fable 5, Anthropic's most capable publicly available model, across eight biomedical benchmarks, four text and four multimodal, using deterministic scoring against fixed answer keys throughout. We include two Claude predecessors and GPT-5 as baselines. Refusal is tracked as a distinct outcome in every result table. That decision produces the paper's central finding. Fable 5 refuses between 8.0% and 99.4% of questions depending on the benchmark, a pattern absent in both predecessors and in GPT-5. Once refused items are excluded from the denominator, Fable 5's accuracy exceeds or meets every other model on every benchmark in this study. We identify two distinguishable refusal patterns: one concentrating in basic-science and mechanism content across MedQA and MedXpertQA MM, confirmed independently on two benchmarks using each benchmark's own category labels; and a separate disease-domain pattern on RareBench, where inborn metabolic disease presentations are refused near-universally while adult-onset autoimmune presentations are not. The primary constraint on Fable 5's biomedical usefulness is willingness to engage, not capability once it does.