Search papers, labs, and topics across Lattice.
This study evaluates the safety performance of five large language models (LLMs) in detecting hate speech across various Urdu scripts and English translations. The findings reveal significant label instability and a concerning 'Missed-in-Urdu' rate, indicating that harmful content is often overlooked in original Urdu scripts compared to their English translations. Notably, smaller open-weight models exhibited higher instability and missed-harm rates, underscoring the inadequacy of current LLM safety evaluations for Urdu, a language with a substantial global speaker base.
Label instability in LLM hate speech detection can reach over 31% when comparing Urdu scripts to English translations, revealing a critical gap in safety evaluation for a major global language.
Urdu, the world's tenth most spoken language with 246 million speakers, remains almost entirely absent from mainstream LLM safety evaluation and nine years of WOAH proceedings. To investigate whether this absence has measurable consequences for content moderation reliability, five large language models, GPT-4o, Claude Sonnet 4.5, Gemini 2.5 Flash, Qwen-2.5, and Llama-3.1, were tested across six datasets spanning Nastaliq Urdu, Roman Urdu, English, and code-switched Urdu-English. Across the five Urdu-script datasets, label instability between original-script and English-translation classification ranged from 15.9% (Gemini 2.5 Flash) to 31.6% (Qwen-2.5), with a'Missed-in-Urdu'rate, content flagged as harmful in English translation but passed as normal in the original script, ranging from 2.4% to 9.9% (median 4.3%). A complete enumeration of all 205 papers across nine ALW/WOAH editions via the ACL Anthology API confirms zero dedicated Urdu papers across the entire period. Results indicate that current LLMs provide uneven safety assurance across Urdu's script varieties, with smaller open-weight models showing substantially higher instability and missed-harm rates than frontier closed models.
