Search papers, labs, and topics across Lattice.
This study systematically investigates gender bias in large language model (LLM)-based fake news detection by augmenting the LIAR benchmark with gender variants of speaker job titles. The analysis reveals that all evaluated LLMs exhibit significant gender sensitivity, with 9.79%-35.13% of statements receiving inconsistent labels based on gender presentation, particularly showing male-skeptic patterns. These findings underscore the detrimental impact of gender bias on the reliability and fairness of automated fact-checking systems, necessitating bias-aware evaluation and mitigation strategies.
Gender bias in LLM-based fake news detection leads to inconsistent judgments, with up to 35% of statements misclassified based on speaker gender.
Large Language Models (LLMs) are increasingly used for automated fact-checking, yet their susceptibility to gender bias in this context remains underexplored. This study presents the first systematic investigation of gender bias in LLM-based fake news detection using real-world data. We augment the LIAR benchmark with three gender variants of speaker job titles (Neutral, Male, Female) for each statement to test whether veracity judgments vary solely based on gender presentation. Six state-of-the-art LLMs are evaluated across multiple bias and fairness metrics. All models exhibit gender sensitivity: 9.79%-35.13% of statements receive inconsistent labels across the three variants, with Male-Female comparisons showing 6.5%-23.6% flip rates. Two primary bias manifestations are identified: instability (inconsistent judgments) and directionality (systematic favoritism). Five models show statistically significant directional effects, with the strongest effects displaying male-skeptic patterns. These findings demonstrate that gender bias undermines both reliability and fairness in LLM-based fake news detection, highlighting the need for bias-aware evaluation and mitigation strategies. The augmented dataset is publicly released to support future research.
