Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates strategies for using LLMs as automatic evaluators across multiple languages, focusing on English, Spanish, and Basque. The authors explore instruction translation, monolingual vs. multilingual fine-tuning, and model size as key factors influencing performance in both in-domain and out-of-domain scenarios. Results show that fine-tuned smaller models can match proprietary models' performance when in-domain data is available, while larger models excel in zero-shot out-of-domain evaluation, providing practical guidelines for building multilingual evaluation pipelines.
Fine-tuning smaller LLMs on in-domain data can rival the performance of larger, proprietary models when used as multilingual judges, challenging the assumption that bigger is always better for text evaluation.
Large language models (LLMs) are increasingly used for the automatic evaluation of generated text, yet most prior work focuses on English. Despite the growing demand for multilingual evaluation, extending LLM-based evaluators to multilingual settings remains challenging, particularly for low-resource languages and scenarios where in-domain data is scarce. This work explores several strategies for developing multilingual LLMs-as-a-judge, considering whether in-domain data is available for fine-tuning or not. We systematically analyze English, Spanish, and Basque, representing high-, mid-, and low-resource languages, considering instruction translation, monolingual versus multilingual supervision, and model size. For evaluation, we extend two existing meta-evaluation datasets to Basque and Spanish. Our results reveal key trade-offs: When in-domain data is available, fine-tuned smaller models can achieve performance comparable to proprietary models, whereas zero-shot evaluation with larger models proves more effective in out-of-domain settings. We also observe that fine-tuning on out-of-domain data can adversely affect model performance. These findings provide practical guidance for building efficient, reliable multilingual evaluation pipelines. The data and code are publicly available at hitz-zentroa/mJudge.