Search papers, labs, and topics across Lattice.
The paper introduces KZ-SafetyPrompts, a new dataset of 5,717 Kazakh prompts designed to evaluate LLM safety across 11 risk categories. The prompts, written in a realistic user query style, aim to expose safety gaps specific to the Kazakh language and cultural context. Baseline evaluations using GPT-4o show a 28.2% overall refusal rate, but significant variance across categories, highlighting the importance of multilingual safety evaluations.
LLM safety evaluations in English miss critical vulnerabilities: GPT-4o's refusal rate on Kazakh safety prompts varies from 5.5% to 53.8% across categories.
Kazakh is underrepresented in resources for evaluating the safety behavior of large language models. We present KZ-SafetyPrompts, a Kazakh prompt dataset for safety evaluation across eleven categories covering common risk areas such as self-harm, violence, child exploitation, sexual content, racist content, radicalization, and regulated goods or illegal activities. The dataset contains 5,717 prompts written natively in Kazakh (Cyrillic), organized by category, with English translations for cross-lingual analysis. Prompts resemble realistic user queries, often in a teen or child style, and are phrased as intent prompts without procedural instructions. We document the writing protocol, labeling procedures (including borderline-case decision rules), and quality-control steps (schema standardization, completeness checks, and deduplication). We also align the categories with widely used safety taxonomies to support integration with existing evaluation pipelines. Baseline results with GPT-4o show an overall refusal rate of 28.2%, varying from 5.5% to 53.8% across categories, indicating that Kazakh prompts expose category-specific safety gaps not captured by English-only evaluation.
