Search papers, labs, and topics across Lattice.
This study evaluates the political fairness of large language models (LLMs) by measuring their perplexity across texts from various political parties. The findings reveal that LLMs exhibit significantly higher perplexity for far-right and nationalist party texts compared to social-democratic party texts, indicating a bias in how these models process political language. Importantly, this bias appears to be rooted in the models' pretraining rather than being mitigated by instruction-tuning, suggesting a systemic issue in LLM training that could affect their application in political contexts.
LLMs show a striking bias, being more perplexed by far-right texts, which raises concerns about their political fairness in real-world applications.
Large Language Models (LLMs) are increasingly used, including in political applications, but their political fairness has been little studied. We assess it using perplexity, posing that a fair model should give equal probability to all political groups. However, we find, across ten LLMs and three datasets covering 37 languages, that LLMs are more perplexed by the texts of far right and nationalist parties than of social-democratic parties. We find this to be consistent with previous work on translation fairness, to the point that perplexity correlates with downstream translation metrics. Our method is applicable to both base LLMs as well as their instruction-tuned counterpart, and we find that both are highly correlated, suggesting that the political fairness of LLMs stems from their pretraining, and is hardly affected by instruction-tuning.
