Search papers, labs, and topics across Lattice.
The authors introduce ChiSafe-PAS, a new human-annotated benchmark dataset of 1,897 adversarial Chinese prompts designed to evaluate LLM safety in high-stakes domains like self-harm, fraud, and illicit trade. The dataset includes detailed annotations such as response labels, obfuscation taxonomy, and risk-level ratings, specifically targeting vulnerabilities arising from Chinese-specific evasion techniques. Experiments using ChiSafe-PAS reveal that safety systems effective in English often fail in Chinese due to linguistic and cultural differences, highlighting the need for culturally grounded safety evaluation.
LLM safety systems that appear robust in English crumble when faced with Chinese-specific adversarial attacks, exposing a critical gap in current alignment strategies.
When Large Language Models (LLMs) are deployed in Chinese-language settings, a troubling pattern emerges: safety systems that work well in English break down. These systems struggle to cross linguistic and cultural bound-aries, leaving models exposed to adversarial prompts that exploit Chinese-specific evasion techniques, including Pinyin romanization, character decomposition, internet slang, and hedging tone. To address this gap, we introduce ChiSafe-PAS (Chinese Safety Pilot Annotation Set), a human-annotated benchmark of 1,897 adversarial Chinese prompts spanning four high-stakes domains: self-harm and violence, drug and illicit trade, fraud, and satire. Of these, 1,544 entries carry complete gold-standard annotations: a 3-class response label (REFUSE, SAFE-REDIRECT, RESPOND), a nine-category obfuscation taxonomy, a risk-level rating, and annotator rationale. We describe the dataset design, annotation process, and obfuscation taxonomy in detail. Our primary goal is practical: to give the research community a high-quality, culturally grounded resource for benchmarking LLM safety alignment. In doing so, we engage three broader tensions in the field: the blurring boundary between training and evaluation data, the need for domain coverage grounded in real-world risk, and the limits of scale as a substitute for cultural expertise.
