Search papers, labs, and topics across Lattice.
The paper introduces GreekMMLU, a new native-sourced benchmark for evaluating LLMs in Greek, comprising 21,805 multiple-choice questions across 45 subjects with difficulty levels spanning primary to professional examinations. The benchmark addresses the lack of authentic Greek evaluation datasets by sourcing questions directly from academic, professional, and governmental exams in Greek. Evaluations of over 80 LLMs using GreekMMLU reveal performance gaps between frontier and open-weight models, and between Greek-adapted and general multilingual models, providing insights for improving LLM capabilities in Greek.
LLMs struggle with native Greek, as evidenced by substantial performance gaps revealed by the new GreekMMLU benchmark, highlighting the need for better adaptation and evaluation methods.
Large Language Models (LLMs) are commonly trained on multilingual corpora that include Greek, yet reliable evaluation benchmarks for Greek-particularly those based on authentic, native-sourced content-remain limited. Existing datasets are often machine-translated from English, failing to capture Greek linguistic and cultural characteristics. We introduce GreekMMLU, a native-sourced benchmark for massive multitask language understanding in Greek, comprising 21,805 multiple-choice questions across 45 subject areas, organized under a newly defined subject taxonomy and annotated with educational difficulty levels spanning primary to professional examinations. All questions are sourced or authored in Greek from academic, professional, and governmental exams. We publicly release 16,857 samples and reserve 4,948 samples for a private leaderboard to enable robust and contamination-resistant evaluation. Evaluations of over 80 open- and closed-source LLMs reveal substantial performance gaps between frontier and open-weight models, as well as between Greek-adapted models and general multilingual ones. Finally, we provide a systematic analysis of factors influencing performance-including model scale, adaptation, and prompting-and derive insights for improving LLM capabilities in Greek.
