Search papers, labs, and topics across Lattice.
This study critically evaluates the effectiveness of human psychometric questionnaires in characterizing LLM behavior by comparing personality profiles derived from Likert self-reports and generation probabilities. The findings reveal a significant divergence between these two profiling methods, with traditional questionnaires leading to socially desirable responses that do not reflect actual LLM behavior in realistic user interactions. The authors argue that generation-based profiling offers a more accurate representation of LLM dispositions, highlighting the limitations of existing psychometric tools in AI contexts.
Human psychometric questionnaires mislead our understanding of LLM behavior, revealing a stark contrast between self-reported and actual response patterns.
We examine whether human psychometric questionnaires can serve as reliable tools for characterizing and predicting LLM behavior in everyday user interactions. We analyze eight open-source LLMs by comparing their value and personality profiles derived from two different methods: Likert self-reports on established questionnaires (PVQ-40/21 and BFI-44/10) and generation probabilities over value-laden responses to everyday user queries. The two profiles diverge substantially. Within-construct item consistency, often cited as evidence of stable LLM dispositions, disappears in generation probabilities. We attribute this gap to the fact that explicit lexical cues in established questionnaire items allow models to recognize the target construct and respond in alignment-consistent, socially desirable ways, whereas realistic user queries provide no such cues. In addition, demographic persona prompts shift models' responses to human questionnaires in ways consistent with real human patterns, but no such shifts appear in the generation probabilities of responses to realistic user queries, showing their limited ability to simulate the behaviors of target demographics in real-world user interactions. Overall, our study shows that human psychometric questionnaires are insufficient tools for predicting LLM behavior and suggests generation-based profiling as a more accurate measure.
