Search papers, labs, and topics across Lattice.
This study evaluates three advanced language models鈥擥PT 5.4, Claude Opus 4.7, and Gemini 3.1 Pro鈥攗sing a challenging, clinician-authored dataset of clinical reasoning tasks across multiple specialties. The results reveal a concerning trend where models struggle significantly with high-priority clinical criteria, achieving pass rates of only 32.4-41.7% for critical tasks, while performing better on lower-stakes criteria. This highlights the gap in current AI capabilities in clinical reasoning and suggests a need for improved training and evaluation methods in this domain.
Despite being frontier models, none of the evaluated LLMs could reliably meet critical clinical reasoning standards, with over half of the essential criteria unmet.
Multiple-choice medical benchmarks are increasingly saturated, and recent rubric-based evaluations such as HealthBench have shown that open-ended clinical performance is far from solved - its"Hard"subset top score remains 32%. We present a small, deliberately difficult evaluation dataset of five clinician-authored clinical scenarios spanning four specialties (anaesthesia, internal/family medicine, emergency medicine, and obstetrics), each accompanied by an atomic, weighted, MECE rubric (25-62 criteria per task; 184 criteria total) authored from a clinician-drafted golden answer. We evaluate three frontier models: GPT 5.4, Claude Opus 4.7, and Gemini 3.1 Pro. Mean rubric pass rates were 0.47 (Claude), 0.39 (GPT), and 0.37 (Gemini). The central finding is an inversion of clinical priority: the highest-weighted (weight-5, critical) criteria passed at only 32.4-41.7%, while low-stakes weight-1 criteria passed at 80-90%. 56 of 108 critical (weight-5) criteria (52%) were satisfied by no model. Three LLM autoraters reproduced expert met/not-met labels on 92.8-94.7% of 552 graded criteria. We position this as a methods-and-preliminary-findings contribution: the five tasks demonstrate a scalable, defensible pipeline ready to develop into a large-scale benchmark.
