Search papers, labs, and topics across Lattice.
This study investigates the impact of social determinants of health (SDoH) on narrative anchoring bias in medical large language models (LLMs) by utilizing a counterfactual dataset, NarrativeShield SDoH MedQA. The evaluation of three instruction-tuned Qwen2.5 models reveals that while the 7B model achieves the highest accuracy and consistency, it still exhibits significant narrative sensitivity errors. These findings underscore the necessity for clinical decision support systems to assess not only the correctness of responses but also their stability across varying patient narratives to enhance trustworthiness.
Medical LLMs can provide the right answers but still falter when faced with different patient narratives, highlighting a critical bias in clinical decision support.
Medical large language models are often judged by how many clinical questions they answer correctly. That view is useful, but it misses a practical risk. A model may know the right answer and still change its response when the same case is written in a different patient voice. This paper evaluates that risk as SDoH aware narrative anchoring bias. We use NarrativeShield SDoH MedQA, a counterfactual medical question answering dataset in which each case appears in persona based narratives while the answer key remains fixed. The dataset is reshaped from wide format into case grouped persona rows. We evaluate three open source instruction tuned LLMs from the Qwen2.5 family: 1.5B, 3B, and 7B. The final experiment uses 300 clinical cases and produces 8,100 model responses across three prompting conditions. We report persona level accuracy, counterfactual consistency, correct consistency, and narrative sensitivity error. Qwen2.5 7B achieves the best accuracy at 56.33 percent and the best correct consistency at 40.33 percent. Paired McNemar exact tests show significant accuracy gains for 7B over 3B in all prompt settings. Even so, narrative sensitivity remains, with the lowest error still at 31.67 percent. These results suggest that trustworthy clinical decision support should be evaluated by both average correctness and stability across medically equivalent patient narratives.
