Search papers, labs, and topics across Lattice.
This study introduces MyoCardBench, a comprehensive benchmark designed to evaluate large language models (LLMs) in the context of real-world cardiovascular care, addressing the limitations of existing benchmarks that often focus on isolated tasks. By utilizing 2,263 items from 13 task-specific datasets and assessing seven LLMs across various clinical dimensions, the research reveals that GPT-5.4 outperformed its peers in both holistic clinical quality and key-point coverage. The findings highlight significant gaps in model performance, particularly in critical areas such as emergency response and treatment planning, underscoring the need for targeted improvements in LLM capabilities for clinical applications.
GPT-5.4 outshines other LLMs in cardiovascular care tasks, but major performance gaps reveal urgent areas for improvement in clinical AI applications.
Background: Most medical large language model (LLM) benchmarks focus on examination knowledge or isolated tasks and may not reflect the longitudinal, multimodal, and safety-critical workflow of cardiovascular care. Objective: To develop MyoCardBench, a real-world benchmark spanning the cardiovascular care continuum, and assess LLM performance across clinical dimensions and specialist tasks. Methods: MyoCardBench includes 2,263 items from 13 task-specific datasets derived from de-identified cardiovascular records and examination data. Sixteen cardiology physicians conducted annotation and reference construction, followed by cross-review from two senior cardiologists. Seven LLMs generated 15,841 outputs under standardized zero-shot settings. Open-ended tasks were evaluated using key-point coverage and holistic clinical quality, while CardioEthics was scored by accuracy. Results: GPT-5.4 achieved the highest macro-average (62.55) and item-weighted mean (62.19), followed by Gemini 3.1 Pro (59.95) and Qwen 3.6 27B (59.72). GPT-5.4 ranked first in all three dimensions. CardioAuxReport performed best (86.38), whereas CardioECGRead (17.25) and CardioEthics (17.34) were lowest. The largest gaps between holistic clinical quality and key-point coverage occurred in CardioComm (52.71), CardioEmergRescue (52.05), and CardioTreatPlan (48.80). Conclusions: To our knowledge, MyoCardBench is the largest real-world, multi-task benchmark for LLM evaluation across the cardiovascular care continuum and offers the broadest coverage of clinically authentic cardiology scenarios reported to date. It provides a rigorous framework for identifying model strengths, clinically important omissions, and priorities for future development.
