Search papers, labs, and topics across Lattice.
This paper introduces P3B3, a benchmark designed to evaluate the performance of Large Language Models (LLMs) in handling conversational prompts across European and Brazilian Portuguese varieties. The study reveals a significant bias in LLMs favoring Brazilian Portuguese (pt-BR) over European Portuguese (pt-PT), with varying degrees of controllability among different models. These findings underscore the importance of addressing linguistic diversity to ensure equitable language representation in AI systems.
Most LLMs exhibit a strong bias towards Brazilian Portuguese, highlighting a critical gap in multilingual representation that could affect communication equity.
As Large Language Models (LLMs) become embedded in everyday communication, capturing regional linguistic variation is essential for reliable and equitable language use. In Portuguese, European (pt-PT) and Brazilian (pt-BR) varieties remain unevenly represented, with pt-BR dominating in data quantity, while LLM preference for Portuguese variants remains underexplored. To address this gap, we introduce P3B3, an expert-curated language variety agnostic benchmark of conversational prompts, along with an evaluation framework for measuring variety bias and controllability. Experiments on several models show that most LLMs exhibit a strong bias toward pt-BR, with variation in controllability across models. These results highlight the need for more balanced multilingual representation across language varieties.
