Search papers, labs, and topics across Lattice.
This study systematically benchmarks eight open-source small language models (SLMs) against commercial models for decision support in emergency departments, focusing on fine-tuning strategies such as zero-shot prompting, prefix tuning, Low-Rank Adaptation (LoRA), and full fine-tuning. The evaluation, conducted on 2,083 MIMIC-IV-ED cases, reveals that LoRA fine-tuned SLMs outperform commercial models in triage level prediction and specialist referral recommendations, although they struggle with diagnosis prediction. Notably, confusion matrix analysis indicates that these fine-tuned SLMs can identify high-severity patients that commercial models often miss, highlighting their potential for clinical application.
LoRA fine-tuned open-source SLMs not only surpass commercial models in critical triage tasks but also uncover high-severity cases that might otherwise go undetected.
Deploying large language models (LLMs) for decision support in emergency departments (EDs) faces two major challenges: privacy risks of transmitting patient data to closed-source commercial LLMs and the lack of systematic evaluation of fine-tuning strategies for locally deployable open-source small language models (SLMs). We benchmarked eight open-source SLMs using zero-shot prompting, prefix tuning, Low-Rank Adaptation (LoRA), and full fine-tuning on three ED tasks: triage level prediction, specialist referral recommendation, and diagnosis prediction. Using 2,083 MIMIC-IV-ED cases and Claude Haiku 4.5 and Claude Sonnet 4.5 as baselines, we found that LoRA fine-tuned open-source SLMs outperform commercial baselines on triage level prediction and specialist referral recommendation, while diagnosis prediction remains challenging for open-source SLMs. Confusion matrix analysis further shows that fine-tuned open-source SLMs can detect highest-severity patients missed by the commercial baselines. These results demonstrate that locally deployable SLMs can achieve clinically competitive performance for ED decision support.
