Search papers, labs, and topics across Lattice.
This paper introduces BavGround, a benchmark designed to evaluate the cultural grounding and dialect competence of large language models (LLMs) specifically for the Bavarian region across English, German, and Bavarian languages. The benchmark consists of 206 multiple-choice questions spanning eight cultural domains, revealing that while strong multilingual models excel overall, they struggle significantly with Bavarian-specific content and localized cultural knowledge. The study highlights the variability in model performance based on evaluation protocols, indicating that different scoring methods can lead to divergent rankings and insights into dialectal understanding.
Multilingual models may excel in general tasks, but they falter significantly when faced with region-specific cultural knowledge, as shown by BavGround's rigorous evaluation.
Cultural evaluation of large language models (LLMs) often focuses on high-resource standard languages, leaving regional culture and dialect communities underrepresented. We introduce BavGround, a benchmark for evaluating Bavarian regional cultural grounding and dialect competence across English, German and Bavarian. BavGround contains 206 multiple-choice source questions across eight cultural domains per language, yielding 618 multi-parallel instances, with items covering both broadly accessible cultural knowledge and source-grounded regional knowledge from journalism, historical sources, and specialist literature. We evaluate fifteen 7B-10B open-weight instruction-tuned models and one closed-model reference. Strong multilingual models perform best overall, but performance drops on Bavarian items and source-grounded questions, indicating persistent difficulty with dialectal and localized cultural knowledge. We further show that conclusions depend strongly on evaluation protocol: raw answer-letter scoring, shuffled-letter scoring, option-text likelihood, generated-answer parsing, and semantic matching can produce different absolute scores and rankings, especially for regionally adapted models. Finally, an exploratory analysis of GENBA-10B checkpoints suggests that continued pretraining improves answer-content likelihood unevenly across domains, while dialect competence remains comparatively weak. BavGround supports localized, protocol-aware evaluation of cultural representation in LLMs.
