Search papers, labs, and topics across Lattice.
This paper replicates and extends existing analyses of dialect-sensitive stereotype generation in LLMs, focusing on differences between Standard American English (SAE) and African-American English (AAE). They evaluate the effectiveness of prompt engineering (role-based and Chain-Of-Thought) and multi-agent architectures (generate-critique-revise) as mitigation strategies across eight prompt templates designed to elicit dialect bias. The study finds that while baseline disparities vary by model, Chain-Of-Thought prompting and multi-agent architectures can effectively mitigate stereotype-bearing differences between SAE- and AAE-related outputs, particularly in adjective and job attribution.
Chain-of-Thought prompting can reduce LLM bias against African-American English, but only if you pick the right model.
Many works in the literature show that LLM outputs exhibit discriminatory behaviour, triggering stereotype-based inferences based on the dialect in which the inputs are written. This bias has been shown to be particularly pronounced when the same inputs are provided to LLMs in Standard American English (SAE) and African-American English (AAE). In this paper, we replicate existing analyses of dialect-sensitive stereotype generation in LLM outputs and investigate the effects of mitigation strategies, including prompt engineering (role-based and Chain-Of-Thought prompting) and multi-agent architectures composed of generate-critique-revise models. We define eight prompt templates to analyse different ways in which dialect bias can manifest, such as suggested names, jobs, and adjectives for SAE or AAE speakers. We use an LLM-as-judge approach to evaluate the bias in the results. Our results show that stereotype-bearing differences emerge between SAE- and AAE-related outputs across all template categories, with the strongest effects observed in adjective and job attribution. Baseline disparities vary substantially by model, with the largest SAE-AAE differential observed in Claude Haiku and the smallest in Phi-4 Mini. Chain-Of-Thought prompting proved to be an effective mitigation strategy for Claude Haiku, whereas the use of a multi-agent architecture ensured consistent mitigation across all the models. These findings suggest that for intersectionality-informed software engineering, fairness evaluation should include model-specific validation of mitigation strategies, and workflow-level controls (e.g., agentic architectures involving critique models) in high-impact LLM deployments. The current results are exploratory in nature and limited in scope, but can lead to extensions and replications by increasing the dataset size and applying the procedure to different languages or dialects.
