Search papers, labs, and topics across Lattice.
This paper introduces the Indi-RomCoM benchmark, designed to evaluate Large Language Models (LLMs) on Romanized Code Mixing (RCM) in Indic-English instructions, addressing a significant gap in their performance evaluation. The study reveals that LLMs struggle with RCM, particularly as the density of code-mixing increases, with notable performance discrepancies between reasoning and detection tasks. The findings highlight the need for improved multilingual capabilities in LLMs, as they consistently underperform in this context compared to monolingual benchmarks.
LLMs falter on Romanized Code Mixing tasks, revealing a critical gap in their multilingual instruction-following abilities.
Romanized Code Mixing (RCM), where bilingual speakers fluidly blend local languages with English in Roman script, has emerged as the dominant form of communication across multilingual communities. While Large Language Models (LLMs) perform strongly on monolingual and native-script benchmarks, their ability to follow instructions and reason over RCM-based content remains largely unexplored. To this end, we introduce the Indi-RomCoM benchmark for facilitating systematic evaluation on Indic Romanized Code-Mixed instructions. Our benchmark spans seven instruction-following tasks, four widely spoken Indic languages, and three controlled code-mixing intensity levels. We extensively evaluate a suite of LLMs covering proprietary, open-weight, and Indic-focused models under zero- and few-shot settings. LLMs consistently underperform on RCM instructions, with performance degrading as code-mixing density increases. Furthermore, reasoning tasks suffer less degradation than detection tasks (e.g., Toxicity) because the generated explanations offer necessary context. We believe Indi-RomCoM helps the community in developing inclusive multilingual systems.
Google Research