Yiming Li

Nanyang Technological University yukunchen@zju.edu.cn; zxy440266@alibaba-inc.com; liyiming.tech@gmail.com Corresponding authors: Xinyu Zhang and Yiming Li. Abstract While large language models (LLMs) have become pivotal to content safety, current evaluation paradigms primarily focus on detecting explicit harms (e.g.e.g., violence or hate speech), neglecting the subtler value dimensions conveyed in digital content. To bridge this gap, we introduce X-Value, a novel Cross-lingual Values Assessment Benchmark designed to evaluate LLMs’ ability to assess deep-level values of content from a global perspective. X-Value consists of more than 5,000 QA pairs across 18 languages, systematically organized into 7 core domains grounded in Schwartz’s Theory of Basic Human Values and categorized into easy and hard levels for discriminative evaluation. We further propose a unique two-stage annotation framework that first identifies whether an issue falls under global consensus (e.g.e.g., human rights) or pluralism (e.g.e.g., religion), and subsequently conducts a multi-party evaluation of the latent values embedded within the content. Systematic evaluations on X-Value reveal that current SOTA LLMs exhibit deficiencies in cross-lingual values assessment (A​c​c<77%Acc<77\%), with significant performance disparities across different languages (Δ​A​c​c>20%\Delta Acc>20\%). This work highlights the urgent need to improve the nuanced, values-aware content assessment capability of LLMs. Our X-Value is available at: https://huggingface.co/datasets/Whitolf/X-Value. Content Warning: This paper contains examples of harmful language. 1 Introduction With the widespread adoption of the internet and the rapid advances in generative artificial intelligence (yang2025qwen3, ; openai2025gpt, ; google2025geminipro, ), digital content is being created and distributed at an unprecedented speed. However, this boom has brought serious challenges: misinformation, cyberbullying, and extremist rhetoric are proliferating across online spaces. Such harmful content not only undermines a healthy digital ecosystem, but also poses threats to individual well-being, social stability, and even national security (mazeika2024harmbench, ; chao2024jailbreakbench, ). Consequently, effective identification and governance of online content to ensure content safety has become an urgent and pivotal issue in the information technology landscape. To alleviate this challenge, large language models (LLMs) have become the dominant tools for content-safety assessment (inan2023llama, ; zhao2025qwen3guard, ; lin2026yufeng, ). Currently, LLM-based evaluation mainly focuses on detecting explicit harmful information (yin2025bingoguard, ; han2024wildguard, ; kumar2025polyguard, ), such as pornography, violence, and hate speech. However, this “minimum-safety” threshold paradigm has clear limitations: it neglects the subtler and more complex values conveyed by the content. As a result, malicious actors can embed implicit stances, contentious examples, or misinformation to spread material that may not be explicitly rule-breaking yet implicitly promotes bias, negativity, or other undesirable values. More importantly, we still lack effective methods to determine whether an LLM’s value judgments are aligned with positive, universal standards of global human values. This raises a central question: how can we rigorously measure an LLM’s ability to assess the deep value dimensions of content? In this paper, we take a global values perspective and, for the first time, introduce a novel Cross-lingual Values Assessment Benchmark (X-Value), as shown in Figure 1. Its core contributions lie in the data composition and the annotation scheme. (1) For data composition, X-Value contains over 5,000 Question-Answer (QA) pairs on global public issues, spanning 18 languages which covers more than 75% of the world’s population, ensuring broad global representativeness. Inspired by Schwartz’s Theory of Basic Human Values (schwartz2012overview, ), the data are systematically organized into seven domain categories aligned with core value dimensions. Moreover, based on the consistency of initial assessment of advanced LLMs, the X-Value is split into easy-level and hard-level subsets for more discriminative evaluation. (2) For annotation scheme, we propose a novel two-stage framework to account for cultural differences: annotators first determine whether the issue involved in the QA pair is global consensus (e.g.e.g., safety and human rights) or pluralism (e.g.e.g., law and religion), and then assess the values of the ‘Answer’ accordingly. Each sample is independently labeled by two native speakers and adjudicated by a third party, ensuring objectivity and reliability. Figure 1: Overview of the X-Value benchmark data composition and annotation scheme. (1) Data Composition: Characterized by linguistic diversity (covering 18 regional languages), an issue domain taxonomy (spanning 7 major global sensitive domains), and difficulty stratification for more discriminative evaluation. (2) Annotation Scheme: A two-stage pipeline consisting of Consensus-Pluralism identification followed by rule-based Values Appropriateness Assessment. Building on the X-Value benchmark, we systematically evaluate the values-oriented content assessment capabilities of representative LLMs. In general, we provide LLMs with the target QA pairs along with human annotation criteria, which incorporates general checkpoints summarized human annotator feedback, to assess content values. Overall experimental results demonstrate that while state-of-the-art (SOTA) LLMs achieve exceptional accuracy (A​c​ceasy>92%Acc_{\text{easy}}>92\%) on the easy-level subset, their performance significantly declines on the hard-level subset (A​c​chard<66%Acc_{\text{hard}}<66\%). We argue that this discrepancy is probably due to the LLMs tending to rely on holistic judgments and failing to identity implicit or localized value conflicts. We further provide a fine-grained analysis of model accuracy across various languages and issue domains. Additionally, experiments on the Qwen3 series reveal a positive correlation between model scale and values assessment capabilities. In summary, our main contributions are two-fold: (1) We introduce X-Value, a novel Cross-lingual Values Assessment Benchmark designed to evaluate the ability of LLMs to assess content values from a global perspective. To the best of our knowledge, this is the first benchmark specifically dedicated to content values assessment. (2) We evaluate eight SOTA LLMs and find that even the most advanced models achieve only approximately 75% accuracy, with significant performance disparities across different languages. Our work highlights the urgent need to improve the capabilities of LLMs in assessing deep-level values for content safety. 2 X-Value Benchmark In this section, we first describe the data composition of X-Value, including data sources, supplement, classification, and selection. We then present our novel two-stage data annotation framework with respect to content values. Finally, we release the dataset and provide an overall analysis. 2.1 Data Composition Sources & Supplement. We collect over 27,000 questions and statements (named Question) 𝒬={qi}i=