Search papers, labs, and topics across Lattice.
This paper investigates the performance of a translation-based pipeline using Aya-expanse-8B for Persian-to-English translation followed by five open-source small language models (SLMs) for extracting 13 clinical features from Persian palliative care transcripts. The study evaluates the SLMs (Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Qwen2.5-1.5B-Instruct, and Gemma-3-1B-it) in a few-shot setting without fine-tuning, finding that Qwen2.5-7B-Instruct achieves the best overall performance (macro-F1: 0.899, MCC: 0.797) and that translation to English generally improves sensitivity and robustness to class imbalance. The research demonstrates the feasibility of using SLMs for clinical information extraction in low-resource languages while preserving privacy.
You can extract clinical features from low-resource languages with surprisingly high accuracy (F1 > 0.89) by pairing a translation model with off-the-shelf small language models, even without fine-tuning.
Extracting clinical information from medical transcripts in low-resource languages remains a significant challenge in healthcare natural language processing (NLP). This study evaluates a two-step pipeline combining Aya-expanse-8B as a Persian-to-English translation model with five open-source small language models (SLMs) -- Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Qwen2.5-1.5B-Instruct, and Gemma-3-1B-it -- for binary extraction of 13 clinical features from 1,221 anonymized Persian transcripts collected at a cancer palliative care call center. Using a few-shot prompting strategy without fine-tuning, models were assessed on macro-averaged F1-score, Matthews Correlation Coefficient (MCC), sensitivity, and specificity to account for class imbalance. Qwen2.5-7B-Instruct achieved the highest overall performance (median macro-F1: 0.899; MCC: 0.797), while Gemma-3-1B-it showed the weakest results. Larger models (7B--8B parameters) consistently outperformed smaller counterparts in sensitivity and MCC. A bilingual analysis of Aya-expanse-8B revealed that translating Persian transcripts to English improved sensitivity, reduced missing outputs, and boosted metrics robust to class imbalance, though at the cost of slightly lower specificity and precision. Feature-level results showed reliable extraction of physiological symptoms across most models, whereas psychological complaints, administrative requests, and complex somatic features remained challenging. These findings establish a practical, privacy-preserving blueprint for deploying open-source SLMs in multilingual clinical NLP settings with limited infrastructure and annotation resources, and highlight the importance of jointly optimizing model scale and input language strategy for sensitive healthcare applications.
