Search papers, labs, and topics across Lattice.
This paper introduces AuAu, a benchmark designed to evaluate the authoritarian tendencies of large language models (LLMs) through a combination of psychometric assessments, contextual behavior vignettes, and realistic user prompts. The evaluation of 17 models from various geopolitical backgrounds reveals that while all models demonstrate significant authoritarian response rates in psychometric tests, these rates decrease in more realistic scenarios. Notably, the study finds that an authoritarian system prompt can manipulate the majority of models to produce increased authoritarian responses, highlighting the urgent need for systematic auditing of AI systems to mitigate such tendencies.
Every tested LLM exhibited authoritarian tendencies, with a single prompt capable of amplifying these biases in 15 out of 17 models.
The worldwide surge of authoritarianism, combined with the increasing central role in users' everyday lives, raises the question of to what extent specific models exhibit or promote authoritarian attitudes and characteristics. We introduce AuAu, a comprehensive benchmark that aims to assess the risk of LLMs generating responses with authoritarian tendencies. This benchmark combines three evaluation approaches: (i) psychometric questions from an extensive pool of 15 human validated instruments; (ii) contextual behavior vignettes probing intended actions in concrete situations; and (iii) responses to realistic user prompts. Unlike prior work, AuAu evaluates not only a general closeness towards authoritarianism but also the established sub-concepts Authoritarian Aggression, Authoritarian Submission, and Conventionalism. Evaluating 17 models from China, the EU, Russia, and the USA, we find that all tested models exhibit substantial authoritarian response rates under the psychometric evaluation, though rates drop significantly in increasingly more realistic downstream task. We further find that an authoritarian system prompt easily manipulates 15 out of 17 models to promote increased authoritarianism. Our results underscore the need for continued, systematic auditing of LLM-based AI systems to detect and ultimately mitigate undesired authoritarian tendencies in generated output. Our code and data are available at: https://github.com/andreaseinwiller/AuAu
