Search papers, labs, and topics across Lattice.
This paper investigates the application of LLM-as-a-Judge in multilingual and low-resource language contexts, revealing significant limitations in proficiency and validation. Analyzing 650 ACL Anthology papers, only 33 addressed these challenges, indicating a troubling overreliance on LLM judgments and inconsistent evaluation outcomes. The authors offer targeted recommendations to improve the robustness of LLM evaluations in these underrepresented linguistic settings.
LLMs may be overtrusted in multilingual evaluations, leading to inconsistent and potentially misleading judgments in low-resource languages.
LLM-as-a-Judge has become the dominant evaluation paradigm for many natural language generation tasks, due to shortcomings of conventional metrics and high correlations with human judgment, albeit mostly in English. There are now attempts to extend LLM-as-a-Judge to multilingual settings including low-resource languages. However, LLMs have limited proficiency in low-resource languages, and there is often no adequate human validation in these settings. To highlight the scope of the problem and current practices, we explore the use of LLM-as-a-Judge evaluators in ACL Anthology papers focusing on multilingual settings and low-resource languages across a diverse set of tasks. Out of 650 papers mentioning LLM-as-a-judge, only 33 of them focus on low-resource or multilingual settings. Our in-depth analysis of these papers indicates inconsistent evaluation outcomes, a tendency to overtrust LLM judgments in multilingual settings, and the widespread reliance on a single judge model per study. To help the NLP community further, we conclude with recommendations about how to use LLM-as-a-Judge in multilingual and low-resource settings.
