Search papers, labs, and topics across Lattice.
This study investigates the cross-lingual skill inconsistency of large language models (LLMs) by employing a multilingual self-play framework, where two instances of the same model compete in a text-based game using different language interfaces. The findings reveal that LLMs exhibit significant variations in performance, with systematic discrepancies in win-loss margins and strategic behaviors across eight languages and six games. Notably, the research highlights that altering the intermediate reasoning language can restore much of the lost performance, underscoring the critical impact of language on LLM decision-making processes.
LLMs can show drastically different skill levels depending on the language used, revealing a hidden barrier to true multilingual capabilities.
Large language models access knowledge inconsistently across languages, but to what extent do they differ in their skill sets when interacting with different languages? This work quantifies cross-lingual skill inconsistency orthogonally from knowledge and general benchmark performance. We do this via multilingual self-play: two instances of the same model compete in a text-based game, each interacting through a different language interface. Since the model, opponent, rules, state space, and available actions remain fixed, this setting isolates the effect of language on the model's realized behavior. We build a multilingual extension to TextArena and evaluate three open-weight models across eight languages and six games covering spatial reasoning, imperfect information, resource allocation, and repeated interaction. We find that the same model can exhibit markedly different playing strength across languages, with systematic variation in win--loss margins, invalid actions, and strategic tendencies. Detailed analyses reveal language-specific failures in spatial reasoning, card-conditioned decisions, and optimal move selection. In some settings, changing only the intermediate reasoning language recovers much of the lost performance, suggesting that language can affect different stages of the decision process. These results show that skill discrepancies are a measurable major roadblock in the development of truly multilingual models. Better understanding these discrepancies can help us design models that perform more equitably across languages.
Google Research