Search papers, labs, and topics across Lattice.
This study critiques traditional closed-ended political stance detection methods in LLMs and introduces a novel approach using fully synthetic prompts generated by LLMs, which are designed to better reflect real-world interactions. By comparing the ecological validity of real, templated, and LLM-generated prompts across contentious policy issues and geopolitical conflicts, the authors find that LLM-generated prompts are perceived as more realistic and convey intended stances more effectively than templated prompts. The research reveals significant discrepancies in stance estimates between templated and LLM-generated prompts, particularly highlighting the overstatement of political leanings in templated formats.
LLM-generated prompts outperform traditional templated prompts in political stance detection, revealing critical biases in existing evaluation methods.
Political stance detection in LLMs has long been dominated by closed-ended, multiple-choice political survey questions---originally designed for humans, and thus lacks the realism and nuance of human-AI interactions in the wild, while also being susceptible to sandbagging. The recent IssueBench framework substantially mitigates these limitations with templated prompts anchored in real-world chat logs. Given the rise in non-work-related use of GenAI assistants, we extend IssueBench beyond writing assistance to include two additional tasks, information seeking and opinion sharing. We argue that templated prompts still lack the nuance of real ones, especially for open-ended tasks, and remain recognisable as evaluation artefacts. We propose the use of fully synthetic (LLM-generated) prompts, produced under detailed instructions with real prompts as seeds. We assess the ecological validity of real, templated, and LLM-generated prompts in a small-scale study covering 3 highly contested policy issues and 3 recent geopolitical conflicts. Human and LLM annotators rank LLM-generated prompts as no less realistic than real ones and clearly more realistic than templated ones, and find that they carry their intended intent and stance more clearly; the LLMs separate templated prompts from the other two far more sharply than the humans do. In a case study, templated and LLM-generated prompts yield systematically different stance estimates for the same model, most visibly under neutral framings, where templated prompts overstate the model's leaning in the direction encoded by the topic-and-stance text (filler) slotted into their templates.
