Search papers, labs, and topics across Lattice.
This paper introduces IndicSafeEval, a novel evaluation framework designed to assess the safety robustness of large language models (LLMs) against persuasive jailbreak attacks in Indian languages. By systematically testing several open-source LLMs with 7,200 adversarial prompts across ten safety-critical categories and six persuasive strategies in languages like Hindi and Bengali, the authors reveal significant disparities in safety performance based on language and phrasing. The findings highlight that current safety evaluations are inadequate, particularly for low-resource languages, and emphasize the necessity for multilingual and persuasion-aware frameworks to better understand LLM vulnerabilities.
Safety evaluations of LLMs vary dramatically across languages, revealing that some harmful content is far more susceptible to manipulation than previously understood.
Large language models (LLMs) are increasingly used in multilingual settings, yet their safety is still evaluated primarily in English. This limits our understanding of how alignment failures manifest in low-resource and culturally diverse languages. We introduce IndicSafeEval, a persuasion-based jailbreak evaluation framework for Indian languages. Our benchmark combines ten safety critical content categories with six human-like persuasive strategies across four different Indian languages, such as Hindi, Bengali, Marathi and Punjabi, resulting in 7,200 adversarial prompts. We conduct a systematic black-box evaluation of several open-source LLMs to examine how their safety behaviour varies across languages, persuasion strategies, and risk categories. Our analysis shows that the model does not behave equally safely across all languages and prompt styles. Instead, safety performance depends strongly on both the languages used and the way a request is phrased using persuasive cues. We further observe that different risk categories exhibit different levels of vulnerability, with some types of harmful content being significantly more susceptible to persuasion-based jailbreaks than others. These findings reveal important limitations of current safety evaluations, which are largely English-centric, and underscore the need for multilingual and persuasion-aware benchmarking frameworks to more accurately assess real-world LLM safety. Our implementation is available at https://github.com/MonSaikat/IndicSafeEval. Warning: this paper contains example data that may be offensive or harmful.
