Search papers, labs, and topics across Lattice.
This paper investigates the impact of persona prompting on the alignment of large language models (LLMs) with human responses in survey predictions, revealing that the selection of persona attributes significantly influences performance. By analyzing various methods for selecting these attributes across multiple social surveys and LLMs, the authors identify conditions under which persona prompting enhances predictive accuracy. The findings indicate that while more attributes do not guarantee better outcomes, understanding human response variation can guide effective persona attribute selection.
The effectiveness of persona prompting in aligning LLMs with human survey responses hinges on strategic attribute selection, not just quantity.
Large Language Models (LLMs) are increasingly used to predict the responses of human participants in survey panels. Towards that goal, persona prompting has recently emerged as a technique to inform and align large pretrained language models. Persona prompting refers to the practice of using short textual descriptions of'personas'in prompts to steer the LLM's generations. Personas describe individuals through different attributes such as their socio-demographics, attitudes, or behaviors, with the aim of aligning LLMs to produce responses that correlate with the corresponding human responses. Yet, recent work has produced mixed and partly conflicting results of persona prompting without clear patterns of success and failure. Among the few consistent findings is that the selection of persona attributes matters, and that using more attributes does not necessarily lead to better performance. It remains unclear how different attribute selection methods perform and how to choose among them. In this paper, we propose that observed human response variation of a survey question is a potential explanation for the mixed performance observed so far. In addition, we compare the performance of persona prompting associated with different methods for selecting persona attributes. We evaluate these methods on four different (general) social surveys across two countries, six LLMs, and twenty prediction tasks per survey. Our work helps to identify when persona prompting can be expected to be useful in survey prediction tasks, and provides new insights on the effectiveness of different attribute selection methods for LLM-based survey prediction using persona prompting.
