Search papers, labs, and topics across Lattice.
The paper introduces IndiKLAR, a new benchmark extending KLAR-CLC to 18 Indian languages and code-mixed variants, to evaluate cross-lingual knowledge consistency in LLMs. Experiments across nine open-weight models reveal a significant accuracy gap between English and native Indian languages (up to 0.50), which is largely closed by using code-mixed inputs. Further analysis using prompting strategies like Translate-in-Thought (TinT) identifies a consistent "flip point" between incorrect and correct predictions across native, code-mixed, and English settings, whether induced by input or internal translation.
LLMs can nail trivia in English, but stumble in Indian languages – unless you throw in some code-mixing, which magically bridges the gap.
Large language models recall knowledge reliably in English but often fail on the same query posed in a lower-resourced language -- a crosslingual consistency gap that remains underexplored for Indian languages and their code-mixed counterparts. To study this gap, we introduce IndiKLAR, an Indic extension of the KLAR-CLC benchmark covering 18 of the 22 scheduled Indian languages and pairing them with code-mixed variants for 11 widely used language pairs, with native-speaker verification of both monolingual and code-mixed variants for these 11 settings. This three-way alignment offers a unique opportunity to examine how knowledge recall consistency varies across the spectrum of English, code-mixed, and native Indian language inputs. Evaluating across nine open-weight models, we find that the native-language accuracy gap to English can reach $\sim$0.50, while code-mixed inputs close most of it -- bringing performance within $\sim$0.05 of English without any model-level intervention. Motivated by this, we evaluate several prompting strategies that vary in how language conversion is exposed, including a two-stage translate-then-answer setup, a one-stage joint translation-and-answer prompt, and Translate-in-Thought (TinT) -- a single-step strategy in which the model converts the input internally and emits only the final answer. Across the performance trajectory native $\rightarrow$ code-mixed $\rightarrow$ English, we identify a consistent flip point -- the boundary between incorrect and correct prediction -- that lies between the native and code-mixed settings. Interestingly, this holds whether the trajectory is induced by the input surface form or by the model's internal conversion process.
Microsoft Research