Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Polistemics, a benchmark designed to evaluate large language models (LLMs) as mediators of political information during elections, addressing the lack of standardized assessment in this critical area. By grounding the evaluation in the concept of Epistemic Modesty, the authors test LLM performance across various informational contexts, revealing that while models perform well with clear evidence, they struggle significantly with vague or contradictory information. The findings indicate that although reliable mediation is possible, current models fail to achieve consistent performance, particularly influenced by party biases and the complexity of political language.
High aggregate scores of LLMs in political mediation hide significant failures, especially under ambiguous conditions, revealing a critical gap in their reliability.
As LLMs increasingly mediate the political information citizens rely on, there is still no standardized way to assess whether they do so responsibly. We introduce Polistemics, a theory-grounded benchmark for evaluating LLMs as mediators of political information in elections. Prior work has treated this task as reproduction rather than mediation, leaving its epistemic dimensions and interaction with imperfect information unaddressed. We ground the evaluation in Epistemic Modesty, a normative standard derived from citizens'epistemic agency, and test it across controlled settings that vary informational properties such as clarity, noise, and consistency. Applying the benchmark to three state-of-the-art LLMs on the 2025 German and Dutch elections, we find that high aggregate scores mask systematic failures. Models mediate reliably under clear evidence but break down under absent, vague, or contradictory information, while flattening the intensity of political language. These failures are likely driven by party priors, influenced by party labels and output language. Reliable mediation appears achievable, but no model delivers it consistently.