Search papers, labs, and topics across Lattice.
This paper introduces C-SafeQA, a comprehensive benchmark designed for evaluating the safety of large language model (LLM) responses in Chinese, addressing the critical distinction between query risk and response policy violations. The benchmark includes 538 base queries and 8,877 adversarial queries, resulting in a dataset of 37,660 labeled query-response records, which are assessed through multi-model adjudication and expert audits. Key findings reveal that unsafe-response rates vary significantly across query types, and the performance of automated safety judges shows notable trade-offs, highlighting specific weaknesses in their evaluation mechanisms.
Unsafe-response rates for Chinese LLMs can soar to over 30% under adversarial conditions, exposing vulnerabilities in automated safety judges.
Safety benchmarks for large language models often assess the risk of a user query, although the outcome of question answering depends on whether the response violates a policy. This distinction is critical in Chinese harmful-content evaluation, where linguistic variation and adversarial transformations can obscure risky intent. We introduce C-SafeQA, a policy-grounded benchmark for response-level Chinese safety evaluation. It comprises 538 base queries and 8,877 adversarial queries answered by four full-model LLM deployments, yielding 37,660 query-response records labeled safe, unsafe, or disputed. Reference labels are generated through agreement-aware multi-model adjudication and blind audits of stratified subsets by three safety experts. C-SafeQA supports both evaluation of target-model safety and auditing of seven automated safety judges against shared reference labels. Unsafe-response rates range from 0.93% to 3.35% on base queries and from 11.68% to 30.05% on adversarial queries. On the adversarial subset, judges show substantial trade-offs between unsafe-response recall and risk-query-conditioned safe-response false positive rate, and no judge dominates all metrics. Both acrostic transformations reduce unsafe recall for all seven judges, revealing mechanism-specific evaluator weaknesses. Dataset records, metadata, verification code, and judge scripts are publicly released to support recomputation, while benchmark construction, target-response generation, and private adjudication remain outside the release boundary.
