Search papers, labs, and topics across Lattice.
This study critiques the reliance on direct translation for multilingual safety evaluations of large language models (LLMs) by constructing culturally-adapted (CA) datasets for Korean, Japanese, Thai, and Khmer. The researchers found that CA prompts significantly improved Attack Success Rates (ASR) and Cultural Realism scores compared to direct translation, revealing that traditional methods underestimate risks in diverse cultural contexts. The results underscore the importance of integrating cultural context into safety evaluations to ensure accurate assessments of LLM performance across different languages.
Culturally-adapted prompts can boost LLM safety evaluations by revealing a 9.3 percentage point increase in attack success rates compared to direct translations.
Multilingual safety evaluation of large language models (LLMs) has predominantly relied on direct translation (DT) of English benchmarks into target languages - an approach that converts surface-level linguistic form while failing to reflect the cultural context embedded in threat scenarios, social norms, and legal frameworks. We construct paired DT and culturally-adapted (CA) datasets via 1:1 seed matching for four languages - Korean (KO), Japanese (JA), Thai (TH), and Khmer (KM) - and compare Attack Success Rate (ASR) and Cultural Realism scores across four open-source LLM. CA prompts yield Delta-ASR > 0 across all 16 language x model combinations (mean +9.3 pp), and DT-based evaluation underestimates risk in 44 of 48 category x language combinations. Language-level analysis reveals that the distribution of threat forms is heterogeneous across languages. Cultural Realism analysis further shows that DT Cultural Depth (C3) scores remain consistently below 1.0 out of 3.0 across all four languages (mean 0.17), whereas CA scores reach up to 2.51, indicating that direct translation produces inputs systematically divergent from those encountered in real-world multicultural settings. These findings demonstrate that adapting benchmarks to language-specific cultural contexts - rather than relying on linguistic translation alone - is necessary for valid multilingual LLM safety evaluation.
