Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study systematically compares the belief-updating processes of large language models (LLMs) and humans in response to persuasive arguments using a natural online persuasion corpus. The findings reveal that LLMs exhibit only slight agreement with human judgments, particularly diverging in their sensitivity to emotional appeals and novel content, which humans find more persuasive. Additionally, the research highlights that LLMs' reliance on topical similarity and formatting over deeper persuasive cues raises concerns about their effectiveness as proxies for human belief updating in social simulations.
LLMs struggle to mimic human belief updating in persuasion, showing only slight agreement and distinct preferences in persuasive cues.
Large language models (LLMs) are increasingly deployed as proxies for human participants in social simulations, yet whether they update their beliefs in response to persuasive arguments, as humans do, remains poorly understood. We conduct a systematic comparison using a naturally occurring online persuasion corpus in which original posters explicitly verify whether a reply changed their view. Our results show that LLMs achieve only slight agreement with humans (Cohen's kappa ranging from 0.079 to 0.178). Content-level analyses show that humans and LLMs agree on the strongest persuasion cues but diverge on finer ones: humans are more swayed by novel content and assertive language, whereas LLMs favor topical similarity and surface-level formatting. At the level of persuasion strategy, LLMs underweight emotional appeals and overweight credibility signals relative to humans, while the type of proposition under debate exerts no measurable effect on the degree of divergence. Furthermore, switching from first-person role-playing to third-person observation shifts all models toward greater resistance to persuasion, with the effect varying across persuasion strategies and textual features. These findings highlight the risk of treating LLM judgments as faithful proxies for human belief updating and point to structural differences in how LLMs and humans process persuasive discourse. Our code is available at https://github.com/tsinghua-fib-lab/LLM-belief-update-cmv.
Tsinghua AI