Search papers, labs, and topics across Lattice.
This study introduces MIRAGE, a comprehensive benchmark designed to evaluate anti-Muslim bias in large language models (LLMs) across three realistic deployment scenarios: direct completion, chain-of-thought reasoning, and agentic decision-making. The findings reveal that chain-of-thought reasoning exacerbates Muslim-violence associations by 12-34% compared to direct completion, while agentic decisions show a significant bias asymmetry favoring non-Muslim cases by 9-22 percentage points. Additionally, the research highlights that bias is influenced by the temporal context of retrieved news, increasing by 18-27% in scenarios involving recent conflicts, underscoring the inadequacy of current mitigation strategies across different conditions.
Chain-of-thought reasoning amplifies anti-Muslim bias in LLMs, revealing critical gaps in existing mitigation strategies across realistic deployment scenarios.
Five years after the discovery of persistent anti-Muslim bias in large language models, most evaluations remain confined to single-turn prompt completion, a setting that no longer reflects how frontier LLMs are deployed. We introduce \textbf{MIRAGE} (Muslim-Identity Reasoning and Agentic Generation Evaluation), a benchmark of 1{,}200 prompts spanning three deployment-realistic conditions: direct completion, chain-of-thought reasoning, and simulated agentic decision-making across content moderation, lending triage, refugee claim summarization, and hiring screens. Across six frontier models, we find that (i) chain-of-thought reasoning \emph{amplifies} rather than suppresses Muslim-violence associations by 12--34\% relative to direct completion, (ii) agentic decisions exhibit a 9--22 percentage-point asymmetry between Muslim and matched non-Muslim cases on identical evidence, and (iii) bias is sharply time-coupled to retrieved news context, increasing 18--27\% under recent-conflict retrieval. Existing prompt-based mitigations transfer poorly across our three conditions, suppressing direct-completion bias while leaving agentic asymmetry largely intact. We release MIRAGE and an open evaluation harness to support targeted mitigation research.
