Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces EuropeMedQA, a new multilingual and multimodal medical examination dataset sourced from official regulatory exams in Italy, France, Spain, and Portugal. The dataset is designed to evaluate the cross-lingual transfer and visual reasoning capabilities of LLMs in a medical context. Initial zero-shot evaluations of contemporary multimodal LLMs on EuropeMedQA reveal performance declines in non-English languages, highlighting the need for more robust multilingual and multimodal medical AI.
LLMs that ace medical exams in English stumble badly when tested on the same knowledge in Italian, French, Spanish, and Portuguese, exposing critical gaps in cross-lingual medical AI.
While Large Language Models (LLMs) have demonstrated high proficiency on English-centric medical examinations, their performance often declines when faced with non-English languages and multimodal diagnostic tasks. This study protocol describes the development of EuropeMedQA, the first comprehensive, multilingual, and multimodal medical examination dataset sourced from official regulatory exams in Italy, France, Spain, and Portugal. Following FAIR data principles and SPIRIT-AI guidelines, we describe a rigorous curation process and an automated translation pipeline for comparative analysis. We evaluate contemporary multimodal LLMs using a zero-shot, strictly constrained prompting strategy to assess cross-lingual transfer and visual reasoning. EuropeMedQA aims to provide a contamination-resistant benchmark that reflects the complexity of European clinical practices and fosters the development of more generalizable medical AI.