Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper conducts a comparative counterfactual audit of ten open-source LLMs on pediatric Emergency Severity Index (ESI) prediction using constructed counterfactual vignettes. The study reveals substantial and inconsistent variation in counterfactual bias across model families, sizes, and domain adaptations, with no clear trend of bias reduction with increased size or medical pretraining. A QLoRA fine-tuned Qwen2.5-7B model demonstrated the lowest sensitivity to counterfactual shifts, suggesting that targeted fine-tuning can mitigate bias more effectively than general scaling or domain pretraining alone.
Open-source LLMs exhibit significant, unpredictable counterfactual bias in clinical triage, with fine-tuning proving more effective than model size or medical pretraining for bias reduction.
Emergency department (ED) triage is a high-stakes prioritization task in which demographic, socioeconomic, and system-context information may improperly influence acuity assignment. Although open-source large language models (LLMs) are increasingly considered for local and privacy-preserving clinical decision support, it remains unclear how counterfactual bias varies across model families, sizes, medical-domain models, and domain-adapted models. We present a comparative counterfactual audit of ten open-source LLMs for pediatric Emergency Severity Index (ESI) prediction. Starting from real and handbook-style clinical vignettes, we construct paired counterfactual variants that change only one injected demographic, socioeconomic, healthcare-access, behavioral, social, or system-context variable while holding the clinical presentation fixed. Models include Qwen2.5-7B, Qwen2.5-14B-Instruct, a QLoRA fine-tuned Qwen2.5-7B, MedGemma variants, MedLLaMA2-7B, GPT-OSS-20B, and GPT-OSS-120B. We measure any counterfactual shift, undertriage, overtriage, shifts greater than one ESI level, mean shift, and mean absolute shift. Counterfactual sensitivity varied substantially and did not consistently decrease with larger model size or medical-domain pretraining. The fine-tuned Qwen2.5-7B showed the lowest overall sensitivity, with a 5.27% any-shift rate and mean absolute shift of 0.0534, versus 16.02% and 0.1706 for the base model. Several larger or medical-domain models showed more significant shifts. Stratified and correlation analyses further revealed clinically important directionality and shared failure patterns hidden by aggregate rates. These findings support counterfactual auditing as a lightweight, clinically interpretable framework for comparing fairness risks in open-source LLMs before clinical deployment.