Search papers, labs, and topics across Lattice.
This study introduces LLM4SDM, the first investigation into open-source smaller language models (OS-sLLMs) for assessing shared decision making (SDM) using the Observer OPTION12 framework, focusing on privacy and sustainability. By evaluating three general-domain and two medical-domain OS-sLLMs on Dutch melanoma consultation transcripts, the research finds that general-domain models significantly outperform their medical counterparts, which struggle with hallucinations and instruction adherence. Notably, Gemma3:12b achieves the highest correlation with human annotations, highlighting the potential of OS-sLLMs in supporting but not replacing human annotators in SDM assessments.
General-domain OS-sLLMs can outperform medical models in assessing shared decision making, despite significant challenges in reasoning and grounding.
We present LLM4SDM, the first study of open-source smaller language models (OS-sLLMs) for automated assessment of shared decision making (SDM) using the Observer OPTION12 framework. Unlike previous work that relies on large commercial models and the shorter OPTION5 instrument, our study focuses on privacy-preserving locally deployable models and Dutch melanoma consultation transcripts. Using expert-annotated clinical consultations, we evaluate three general-domain and two medical-domain OS-sLLMs during a development-phase pilot study. Results show that general-domain models outperform medical-domain models, which exhibit substantial hallucination and instruction-following failures. Gemma3:12b achieves the strongest agreement with human annotations (Pearson r=0.51, Spearman \r{ho}=0.59). Item-level and qualitative analyses reveal systematic challenges related to temporal discourse reasoning, conversational role attribution, and evidence grounding. We further introduce a Judge-LLM consensus framework designed to support disagreement resolution among multiple models. Our findings suggest that while current OS-sLLMs cannot replace human annotators, they offer a promising foundation for privacy-preserving human-in-the-loop SDM assessment.
