Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Polar, a comprehensive benchmark designed to evaluate political bias in large language models (LLMs) using option-level likelihoods across diverse political and linguistic contexts. The benchmark consists of 4,026 instances covering two ideological axes and eight issue categories, revealing systematic variations in bias among 38 LLMs based on political context, issue category, and presentation language. Notably, while all models exhibit a left-progressive bias in U.S. contexts, they demonstrate more nuanced and varied biases in South Korean contexts, underscoring the importance of multilingual evaluations in understanding LLM behavior.
Political bias in LLMs is not only context-dependent but can shift dramatically based on the language used for presentation.
Political bias in large language models (LLMs) is increasingly significant, but difficult to measure reproducibly across political and linguistic contexts. We introduce Polar, a 4,026-instance multiple-choice benchmark that measures political bias through option-level likelihoods rather than prompt-based generation. Polar covers two ideological axes and eight issue categories derived from the Manifesto Project, and evaluates models in parallel across U.S. and South Korean political contexts. Across 38 LLMs, measured bias varies systematically with political context, issue category, model group, and presentation language. All models lean left-progressive on U.S. political content, but show more centered and mixed patterns on South Korean content. Translation experiments further show that presentation language alone can shift measured bias. These findings highlight the need for multilingual and cross-contextual evaluation of political bias in LLMs.