Search papers, labs, and topics across Lattice.
This paper introduces ClinEnv, an interactive benchmark designed to evaluate large language models (LLMs) as attending physicians in a realistic clinical setting, focusing on the complexities of decision-making in healthcare. By employing a Longitudinal Inpatient Simulation framework, ClinEnv assesses both the quality of decisions made by the models and their information-gathering processes through a structured sequence of decision stages. The results reveal that even the best-performing model achieves only a 0.31 decision F1 score, highlighting a significant disparity between outcome quality and the quality of the decision-making process, particularly in management decisions.
ClinEnv reveals that LLMs struggle significantly with management decisions in clinical scenarios, achieving only 0.17 F1 for these critical actions despite better performance in diagnosis.
Clinical practice is not the selection of an answer from enumerated options: a physician gathers heterogeneous information incrementally and commits to sequential, irreversible decisions under uncertainty. Static benchmarks cannot probe and existing interactive medical benchmarks each compromise on at least one of them. We present ClinEnv, an interactive benchmark that evaluates LLMs as attending physicians over real inpatient admissions under a paradigm we term Longitudinal Inpatient Simulation. Each case is automatically constructed into an ordered sequence of decision stages; at every stage the model must actively query four specialized agents before committing to medications, procedures, and diagnoses. ClinEnv scores both what the model decides, through deterministic ontology-grounded matching, and how it gathers information. Across seven models, the strongest reaches only 0.31 decision F1, and outcome quality is sharply decoupled from process quality. Difficulty concentrates in management decisions and later stages, where models recover discharge diagnoses far more reliably than management actions (0.51 vs. 0.17 F1) and continue to issue redundant queries as cases progress. ClinEnv makes this information-acquisition gap, invisible to outcome-only evaluation, directly measurable.
Tsinghua AI