Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the ability of large language models (LLMs) to maintain consistent stances on contentious societal issues using a new dataset called HardChoices. The findings reveal that LLMs exhibit a surprising lack of neutrality and often agree on positions, despite the inherent disagreements among humans within the same ideological groups. This raises important questions about the coherence and reliability of LLMs' responses to complex societal dilemmas, highlighting potential limitations in their capacity for nuanced understanding.
LLMs display surprising agreement on contentious issues, challenging assumptions about their ability to reflect diverse human perspectives.
A lot of research attention has been devoted to checking whether large language models (LLMs) are politically biased. This work has largely focused on high-level ideological dimensions, such as left--right or progressive--conservative, and it has been shown that while LLMs are predominantly left and progressive leaning, largely mimicking the biases in the training data, they can be to some extent steered to change their preferences in post-training. In this short note, we check if LLMs have robust stances with regard to major substantive societal issues, on which members of the same ideological camp are often in disagreement, summarised in a novel dataset \textsc{HardChoices}. We show that, faced with this line of questioning, LLMs, both large and small, surprisingly rarely declare neutrality, are often incoherent, and demonstrate a remarkable degree of agreement on issues where they do take stances.