Search papers, labs, and topics across Lattice.
The paper introduces CROSS, a new benchmark to evaluate cultural safety reasoning in Large Vision-Language Models (LVLMs) across 16 countries and 14 languages, addressing the gap in existing benchmarks that primarily focus on physical safety. The authors evaluate 21 leading LVLMs using CROSS-Eval, a framework based on intercultural theory, revealing significant cultural safety gaps in awareness and compliance. They then propose and demonstrate the effectiveness of supervised fine-tuning and preference tuning strategies to improve cultural awareness and compliance in GPT-4o, while maintaining general multimodal capabilities.
LVLMs struggle to navigate cultural nuances, with even the best models achieving only 62% awareness and 38% compliance on a new benchmark spanning 16 countries.
Large vision-language models (LVLMs) are increasingly deployed in globally distributed applications, such as tourism assistants, yet their ability to produce culturally appropriate responses remains underexplored. Existing multimodal safety benchmarks primarily focus on physical safety and overlook violations rooted in cultural norms, which can result in symbolic harm. To address this gap, we introduce CROSS, a benchmark designed to assess the cultural safety reasoning capabilities of LVLMs. CROSS includes 1,284 multilingual visually grounded queries from 16 countries, three everyday domains, and 14 languages, where cultural norm violations emerge only when images are interpreted in context. We propose CROSS-Eval, an intercultural theory-based framework that measures four key dimensions: cultural awareness, norm education, compliance, and helpfulness. Using this framework, we evaluate 21 leading LVLMs, including mixture-of-experts models and reasoning models. Results reveal significant cultural safety gaps: the best-performing model achieves only 61.79% in awareness and 37.73% in compliance. While some open-source models reach GPT-4o-level performance, they still fall notably short of proprietary models. Our results further show that increasing reasoning capacity improves cultural alignment but does not fully resolve the issue. To improve model performance, we develop two enhancement strategies: supervised fine-tuning with culturally grounded, open-ended data and preference tuning with contrastive response pairs that highlight safe versus unsafe behaviors. These methods substantially improve GPT-4o's cultural awareness (+60.14%) and compliance (+55.2%), while preserving general multimodal capabilities with minimal performance reduction on general multimodal understanding benchmarks.
