Search papers, labs, and topics across Lattice.
The paper introduces Indic Jailbreak Robustness (IJR), a novel judge-free benchmark to evaluate the adversarial safety of LLMs across 12 Indic and South Asian languages using both contract-bound (JSON) and naturalistic (Free) prompt formats. The benchmark reveals that contract-bound prompts inflate refusal rates without preventing jailbreaks, English-to-Indic attacks transfer effectively, and orthography (romanization) significantly impacts jailbreak success rates. The study highlights the limitations of English-centric, contract-focused evaluations and the increased vulnerability of South Asian language users.
LLMs are shockingly vulnerable to jailbreaks in South Asian languages, with models reaching 1.0 Jailbreak Success Rate in free-form prompts, even after inflated refusal rates in contract-based settings.
Safety alignment of large language models (LLMs) is mostly evaluated in English and contract-bound, leaving multilingual vulnerabilities understudied. We introduce \textbf{Indic Jailbreak Robustness (IJR)}, a judge-free benchmark for adversarial safety across 12 Indic and South Asian languages (2.1 Billion speakers), covering 45216 prompts in JSON (contract-bound) and Free (naturalistic) tracks. IJR reveals three patterns. (1) Contracts inflate refusals but do not stop jailbreaks: in JSON, LLaMA and Sarvam exceed 0.92 JSR, and in Free all models reach 1.0 with refusals collapsing. (2) English to Indic attacks transfer strongly, with format wrappers often outperforming instruction wrappers. (3) Orthography matters: romanized or mixed inputs reduce JSR under JSON, with correlations to romanization share and tokenization (approx 0.28 to 0.32) indicating systematic effects. Human audits confirm detector reliability, and lite-to-full comparisons preserve conclusions. IJR offers a reproducible multilingual stress test revealing risks hidden by English-only, contract-focused evaluations, especially for South Asian users who frequently code-switch and romanize.
