Search papers, labs, and topics across Lattice.
This paper benchmarks 17 LLMs as zero-shot annotators for Bangla hate speech detection, a low-resource and identity-sensitive task. The study reveals annotator bias and instability in LLM judgments, challenging the assumption that larger models consistently outperform smaller ones in annotation quality. The findings demonstrate that task-aligned smaller models can exhibit more consistent behavior, highlighting the need for careful evaluation before deploying LLMs for sensitive annotation tasks in low-resource languages.
Bigger isn't always better: smaller, task-aligned LLMs can exhibit more consistent annotation behavior than their larger counterparts in sensitive, low-resource settings like Bangla hate speech detection.
Large Language Models (LLMs) are increasingly used as automated annotators to scale dataset creation, yet their reliability as unbiased annotators--especially for low-resource and identity-sensitive settings--remains poorly understood. In this work, we study the behavior of LLMs as zero-shot annotators for Bangla hate speech, a task where even human agreement is challenging, and annotator bias can have serious downstream consequences. We conduct a systematic benchmark of 17 LLMs using a unified evaluation framework. Our analysis uncovers annotator bias and substantial instability in model judgments. Surprisingly, increased model scale does not guarantee improved annotation quality--smaller, more task-aligned models frequently exhibit more consistent behavior than their larger counterparts. These results highlight important limitations of current LLMs for sensitive annotation tasks in low-resource languages and underscore the need for careful evaluation before deployment.
