Search papers, labs, and topics across Lattice.
This paper introduces the Dual-Metric Bias Assessment (DMBA) framework to evaluate social biases in LLMs within the underrepresented Nepali cultural context, using a dataset of 2400+ stereotypical and anti-stereotypical sentence pairs. The DMBA framework combines explicit agreement with biased statements and implicit stereotypical completion tendencies to reveal biases in seven state-of-the-art LLMs. Results indicate that while models exhibit measurable explicit agreement bias, implicit completion bias follows a non-linear relationship with temperature and is not well-predicted by agreement metrics, highlighting the need for culturally grounded debiasing strategies.
LLMs show strong implicit biases in underrepresented cultural contexts like Nepal, and these biases are poorly captured by standard agreement metrics, demanding new evaluation paradigms.
Large language models (LLMs) increasingly influence global digital ecosystems, yet their potential to perpetuate social and cultural biases remains poorly understood in underrepresented contexts. This study presents a systematic analysis of representational biases in seven state-of-the-art LLMs: GPT-4o-mini, Claude-3-Sonnet, Claude-4-Sonnet, Gemini-2.0-Flash, Gemini-2.0-Lite, Llama-3-70B, and Mistral-Nemo in the Nepali cultural context. Using Croissant-compliant dataset of 2400+ stereotypical and anti-stereotypical sentence pairs on gender roles across social domains, we implement an evaluation framework, Dual-Metric Bias Assessment (DMBA), combining two metrics: (1) agreement with biased statements and (2) stereotypical completion tendencies. Results show models exhibit measurable explicit agreement bias, with mean bias agreement ranging from 0.36 to 0.43 across decoding configurations, and an implicit completion bias rate of 0.740-0.755. Importantly, implicit completion bias follows a non-linear, U-shaped relationship with temperature, peaking at moderate stochasticity (T=0.3) and declining slightly at higher temperatures. Correlation analysis under different decoding settings revealed that explicit agreement strongly aligns with stereotypical sentence agreement but is a weak and often negative predictor of implicit completion bias, indicating generative bias is poorly captured by agreement metrics. Sensitivity analysis shows increasing top-p amplifies explicit bias, while implicit generative bias remains largely stable. Domain-level analysis shows implicit bias is strongest for race and sociocultural stereotypes, while explicit agreement bias is similar across gender and sociocultural categories, with race showing the lowest explicit agreement. These findings highlight the need for culturally grounded datasets and debiasing strategies for LLMs in underrepresented societies.
