Search papers, labs, and topics across Lattice.
This paper introduces Schützen, a safety evaluation dataset specifically designed for assessing large language models (LLMs) in the Bulgarian and German contexts, addressing the existing bias towards English and Chinese in safety evaluations. Through experiments with multilingual and language-specific LLMs, the authors uncover significant cross-language differences in safety behavior, underscoring the importance of context-aware evaluation resources. The findings emphasize the need for tailored safety assessments to mitigate risks associated with LLM deployment in diverse sociocultural environments.
Cross-language safety evaluations reveal that LLMs exhibit starkly different risk profiles in Bulgarian compared to German, challenging the notion of universal model safety.
Large language models are increasingly deployed across professional domains, bringing hard-to-predict risks, including the generation of harmful or disrespectful content. Although substantial progress has been made in developing safety evaluation datasets, existing resources remain overwhelmingly English- and Chinese-centric. This limitation is particularly pronounced when evaluating languages that operate within shared sociocultural, legal, and ethical contexts. To address this gap, we introduce Schützen: a German--Bulgarian safety dataset designed to assess model answerability under risk, covering both a low-resource language (Bulgarian) and a high-resource language (German). Experiments with multilingual and language-specific LLMs reveal pronounced cross-language differences in safety behavior, highlighting the necessity of tailored, region-specific evaluation resources to support the responsible deployment of LLMs in Germany and Bulgaria. Datasets and code are available at https://github.com/xnlp-lab/Schutzen. Warning: this paper contains examples that may be offensive, harmful, or biased.
