Search papers, labs, and topics across Lattice.
This study introduces MedSP1000, an interactive benchmark designed to evaluate large language models (LLMs) in dynamic clinical decision-making using standardized patient cases. By simulating real clinical encounters, the evaluation reveals that performance on traditional static benchmarks does not correlate with success in these complex scenarios, with the top-performing model achieving only 60.4% of expert-defined rubric items. These findings highlight the inadequacy of current LLMs for safe integration into clinical practice and underscore the importance of process-level evaluations to identify critical failure modes.
Static benchmarks fail to predict LLM performance in dynamic clinical settings, with top models only achieving 60.4% of expert criteria in real-world simulations.
Large language models (LLMs) are increasingly proposed as clinical agents, yet static, single-turn benchmarks cannot capture how a model dynamically delivers care across an encounter: gathering information, planning treatment, and adapting longitudinal management across successive patient states. Medical education has long addressed an analogous challenge through standardized patients (SPs): trained actors who consistently portray clinical cases, enabling realistic practice and objective, scripted assessment. Here we introduce MedSP1000, an SP-derived interactive benchmark for clinical-agent evaluation, including 1,638 SP cases with 24,602 trajectory-level peer-reviewed rubrics. MedSP1000 converts peer-reviewed SP teaching cases into executable scenarios with defined SP case scripts, clinical environment contexts, and human-validated structured rubric. In each simulation evaluation run, a clinical agent interacts in closed loop with a patient agent and an environment controller, and its behaviour is scored throughout the encounter against expert criteria specified in the original materials. Applying MedSP1000 to a range of general-purpose and medically specialized LLMs, we find that performance on static benchmarks does not reliably translate to such educational scenarios. The best-performing model, GPT-5.5, completes only 60.4% of expert-defined rubric items, whereas the strongest medically specialized model reaches 40.0%; increasing test-time compute produces no measurable gain. These results suggest that current LLMs, including agentic systems tuned for medicine, are not yet reliable enough to be safely integrated into actual clinical practice. More broadly, MedSP1000 shows how process-level, SP-style evaluation can reveal clinically relevant failure modes that single-turn benchmarks miss.
