Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study systematically evaluates the performance of large language models (LLMs) in multi-turn dialogue games across 30 languages, including all 24 official EU languages. The findings reveal that commercial LLMs consistently outperform open-weight models, with the latter failing to achieve satisfactory performance in any official EU language. Notably, while linguistic parity is possible, it cannot be achieved solely through public web text, as evidenced by the cost and performance disparities observed across languages.
Commercial LLMs outperform open-weight models across all EU languages, revealing significant performance and cost disparities in multilingual applications.
We evaluate large language models (LLMs) as language agents playing goal-directed dialogue games in self-play across 30 languages: the 24 official EU languages plus six others. Unlike static or preference-based evaluation, this paradigm is multi-turn, reference-free and programmatically scored, and because the game mechanics are language-agnostic it extends to a new language by localising a fixed set of prompt and word-list files. Evaluating nine open-weight and commercial LLMs, we find that no open-weight model covers the EU-24 well: in every official language both commercial systems outscore every open-weight model, and the two weakest average below 40 points across the EU-24. The commercial systems stay ahead even in languages with four orders of magnitude less public web text, showing that linguistic parity is achievable, but not from public crawls alone. A model's home region lifts it without closing the gap: Chinese is the strongest of all 30 languages for two Chinese-developed models, yet the best Chinese score of any model belongs to a US commercial system. Coverage is also not parity of service. Pooled over models and languages, the median non-English language costs 31% more to run than English, and scores 10% lower.