Search papers, labs, and topics across Lattice.
The paper introduces AlbanianLLMSafety, the first publicly available dataset for evaluating LLM safety in Albanian, a low-resource language. The dataset comprises 2,951 prompts across 11 safety categories, each with Albanian text, English translations, and detailed labels. This resource enables targeted safety evaluations, fine-tuning, and red-teaming efforts for Albanian-speaking communities, addressing a critical gap in multilingual LLM safety.
LLM safety evaluations now have a crucial resource for Albanian, a significantly underrepresented low-resource language.
Safety evaluation of Large Language Models (LLMs) has largely focused on high-resource languages, leaving low-resource languages critically underserved. We present AlbanianLLMSafety, the first publicly available safety evaluation dataset for LLMs in Albanian, a linguistically distinct low-resource language with approximately 7.5 million speakers across Albania, Kosovo, North Macedonia, and the diaspora. The dataset contains 2,951 prompts spanning 11 safety categories, including self-harm, violence, racist content, child exploitation, and radicalization, with an average of 268 prompts per category. Each prompt is provided in Albanian with an English reference translation and a detailed category label. This resource addresses a significant gap in safety evaluation infrastruc-ture for low-resource languages and provides an essential benchmark for developing safer, more inclusive LLMs. The dataset will be provided upon request to support safety evaluation, fine-tuning, red-teaming, and guardrail development for Albanian-speaking communities.
