Search papers, labs, and topics across Lattice.
This paper compares general and clinical LLMs, including a newly introduced Marmoka family of lightweight 8B-parameter clinical LLMs, on medical question answering tasks in both English and Spanish, using a novel perturbation-based evaluation benchmark. Results indicate that clinical LLMs do not consistently outperform general-purpose models on English tasks and exhibit limitations in instruction following, while Marmoka models show promise for low-resource languages like Spanish. The study concludes that current benchmarks may not adequately assess genuine medical expertise in LLMs.
Domain-specific LLMs might be overrated: clinical LLMs barely edge out general models on medical benchmarks, suggesting current evaluations miss the mark.
BACKGROUND: Recent studies have shown that domain-adapted large language models (LLMs) do not consistently outperform general-purpose counterparts on standard medical benchmarks, raising questions about the need for specialized clinical adaptation. METHODS: We systematically compare general and clinical LLMs on a diverse set of multiple choice clinical question answering tasks in English and Spanish. We introduce a perturbation based evaluation benchmark that probes model robustness, instruction following, and sensitivity to adversarial variations. Our evaluation includes, one-step and two-step question transformations, multi prompt testing and instruction guided assessment. We analyze a range of state-of-the-art clinical models and their general-purpose counterparts, focusing on Llama 3.1-based models. Additionally, we introduce Marmoka, a family of lightweight 8B-parameter clinical LLMs for English and Spanish, developed via continual domain-adaptive pretraining on medical corpora and instructions. RESULTS: The experiments show that clinical LLMs do not consistently outperform their general purpose counterparts on English clinical tasks, even under the proposed perturbation based benchmark. However, for the Spanish subsets the proposed Marmoka models obtain better results compared to Llama. CONCLUSIONS: Our results show that, under current short-form MCQA benchmarks, clinical LLMs offer only marginal and unstable improvements over general-purpose models in English, suggesting that existing evaluation frameworks may be insufficient to capture genuine medical expertise. We further find that both general and clinical models exhibit substantial limitations in instruction following and strict output formatting. Finally, we demonstrate that robust medical LLMs can be successfully developed for low-resource languages such as Spanish, as evidenced by the Marmoka models.
