Search papers, labs, and topics across Lattice.
This study introduces a systematic auditing framework to evaluate the political preferences expressed by Large Language Models (LLMs) in the context of Italian political parties and leaders. By analyzing LLM responses across nine criteria, the research reveals significant variations in evaluations based on model differences, prompt formulations, and adopted personas, rather than attempting to discern the models' "true" beliefs. The findings highlight the influence of LLMs on political attitudes and underscore the importance of understanding their behavior during election periods.
LLMs exhibit striking inconsistencies in political evaluations, influenced by prompt design and model persona, raising critical questions about their role in shaping public opinion during elections.
As users increasingly turn to Large Language Models (LLMs) for information and advice on political matters, particularly during election periods, the political preferences expressed by these systems have become a matter of public interest. Prior research has shown that interactions with LLMs can influence users'political attitudes and choices, raising questions about how these models themselves evaluate political actors. In this paper, we investigate whether and how LLMs express preferences toward political parties and political leaders. We introduce a systematic and reproducible auditing framework in which multiple LLMs are prompted to evaluate parties and leaders across nine criteria. Rather than attempting to infer the models'"true"political beliefs, we focus on their observable behavior, examining consistency across evaluations, differences between models, refusal rates, and sensitivity to prompt formulation. We further investigate how these evaluations vary when models are instructed to adopt different personas. We demonstrate the framework through an Italian case study, providing a systematic analysis of LLM-generated political evaluations on italian parties and leaders.
