Search papers, labs, and topics across Lattice.
DeepMindANLArtifex LabsChunghwa Telecom LaboratoriesIIT MadrasIMDAPoly MontrealRochester Institute of TechnologySNUYonseiJul 7, 2026arXiv:2607.06196This paper introduces Pluralis v0.1, a novel multimodal, multi-regional, and multilingual dataset designed to address the shortcomings of existing AI safety benchmarks that are predominantly Western-centric. By sourcing 6,448 prompts from six Asia-Pacific countries and eight languages, the dataset emphasizes localized safety hazards and cultural appropriateness, which are often overlooked in traditional evaluations. The study reveals that Vision-Language Models (VLMs) exhibit locale-specific failure modes, highlighting the need for culturally aware AI assessments that can better inform global deployments.
VLMs are prone to critical failures that vary significantly across cultures, exposing the inadequacy of Western-centric safety benchmarks.
Current AI safety evaluation and benchmarking frameworks predominantly rely on Western-centric culture-agnostic defaults that mask critical regional laws, socio-linguistic nuances, and cultural taboos, leaving Vision-Language Models (VLMs) vulnerable in global deployments. We introduce Pluralis v0.1: a novel multimodal, multi-regional, and multilingual dataset built from a culture-first perspective. Spanning 6,448 prompts across six Asia-Pacific countries (Bangladesh, India, Korea, Pakistan, Singapore, Taiwan) and eight languages, Pluralis diverges from prior work by natively sourcing localized safety hazards rather than adapting Western datasets. Crucially, it introduces a multimodal evaluation paradigm: user text (e.g.,"Should I gift this?") and an image referring to"this"(e.g., a clock) - both innocuous in isolation, but synergistically triggering specific legal or cultural violations. Pluralis disentangles universal safety violations from localized cultural appropriateness, establishing the latter as a first-class evaluation axis. To operationalize this, we present Judge-Pluralis, an agreement-gated LLM-as-a-Judge ensemble trained on examples classified in an empirically derived cultural taxonomy. Observing VLM behavior on a subset of the Pluralis surfaces recurring, locale-specific failure modes such as image misidentifications with downstream harm, missed item-context-locale interactions, and inadequate refusals. These failure modes vary systematically across locales and languages, exposing blind spots that globally averaged metrics conceal. Ultimately, Pluralis is not presented as a solved evaluation framework for cultural alignment, but rather as a first step and catalyst for future innovation. We call upon the research community to utilize this foundation to advance the science of multilingual, multicultural evaluation to better support AI cultural alignment globally.
