Search papers, labs, and topics across Lattice.
This paper introduces EHR-Complex, a benchmark designed to evaluate the capabilities of clinical agents in complex EHR analysis, addressing the limitations of existing benchmarks that rely on simplified SQL generation. Built on the extensive MIMIC-IV dataset, EHR-Complex features 52K tasks that require agents to perform interactive database reasoning, revealing significant challenges in executing SQL queries and Python code for clinical intents. Evaluation shows that even the best-performing models struggle with a mere 62.3% accuracy and exhibit substantial fragility, highlighting critical areas for improvement in EHR reasoning capabilities.
Clinical agents face a staggering 62.3% accuracy ceiling in complex EHR reasoning, revealing significant gaps in current AI capabilities.
Clinical agents promise to democratize access to electronic health records (EHRs), yet existing benchmarks fail to reflect the complexity of practical EHR analysis, e.g., often operating on idealized, clean EHRs via static SQL generation rather than interactive execution. In this work, we introduce EHR-Complex, a large-scale benchmark designed for interactive clinical database reasoning. Built on the large MIMIC-IV substrate (365K patients, 31 tables, 500M+ records), EHR-Complex comprises about 52K tasks spanning six clinical intents, supporting both patient-level and population-level queries, where each task requires an agent to interact with a sandboxed environment by executing SQL queries or Python code. Notably, EHR-Complex considers the real-world SQL task complexity for longitudinal multi-table aggregation and compositional reasoning, resulting in 31.93 SQL structural components per query on average. Evaluation results on EHR-Complex reveal the clinical difficulty of these EHR reasoning scenarios, with the top-performing model achieving only 62.3% exact-match accuracy. Pass^k consistency drops below 50% for nearly all evaluated models at k=4, exposing broad stochastic fragility. A fine-grained analysis of more than 3,800 failed trajectories for representative LLMs reveals three dominant failure modes: SQL logic errors, medical-code lookup failures, and semantic misunderstandings. EHR-Complex provides a rigorous testbed for clinical agents and highlights remaining gaps in robust reasoning for large-scale EHR analysis.
