Search papers, labs, and topics across Lattice.
The paper introduces UbuntuGuard, a new policy-based safety benchmark for African languages, designed to address the limitations of existing Western-centric guardian models. They created the benchmark using adversarial queries authored by 155 domain experts across sensitive fields to derive context-specific safety policies and reference responses. Evaluations of 13 models (general-purpose LLMs and guardian models) demonstrate that current benchmarks overestimate multilingual safety and that cross-lingual transfer is insufficient, highlighting the need for culturally grounded safety benchmarks.
Current safety benchmarks fail dramatically when applied to African languages, necessitating culturally-grounded benchmarks like UbuntuGuard to avoid overestimating multilingual safety.
Current guardian models are predominantly Western-centric and optimized for high-resource languages, leaving low-resource African languages vulnerable to evolving harms, cross-lingual safety failures, and cultural misalignment. Moreover, most guardian models rely on rigid, predefined safety categories that fail to generalize across diverse linguistic and sociocultural contexts. Robust safety, therefore, requires flexible, runtime-enforceable policies and benchmarks that reflect local norms, harm scenarios, and cultural expectations. We introduce UbuntuGuard, the first African policy-based safety benchmark built from adversarial queries authored by 155 domain experts across sensitive fields, including healthcare. From these expert-crafted queries, we derive context-specific safety policies and reference responses that capture culturally grounded risk signals, enabling policy-aligned evaluation of guardian models. We evaluate 13 models, comprising six general-purpose LLMs and seven guardian models across three distinct variants: static, dynamic, and multilingual. Our findings reveal that existing English-centric benchmarks overestimate real-world multilingual safety, cross-lingual transfer provides partial but insufficient coverage, and dynamic models, while better equipped to leverage policies at inference time, still struggle to fully localize African-language contexts. These findings highlight the urgent need for multilingual, culturally grounded safety benchmarks to enable the development of reliable and equitable guardian models for low-resource languages. Our code can be found online.\footnote{Code repository available at https://github.com/hemhemoh/UbuntuGuard.
