Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study conducts a comprehensive psychometric evaluation of large language models (LLMs) as synthetic survey respondents by analyzing their ability to preserve the joint distribution and latent structures of real human data. Using a Lithuanian organizational-psychology dataset and a robust lineup of 37 models from various AI organizations, the authors find that while LLMs replicate the qualitative relationships found in human responses, they ultimately fail to match human reliability and demographic effects. The key result reveals that LLMs are more similar to each other than to actual human respondents, with significant discrepancies in predictive validity and mediation pathways, indicating that LLMs cannot serve as reliable substitutes for human survey data.
LLMs exhibit a striking similarity to each other but fail to capture the nuanced psychometric properties of human responses, raising questions about their use in survey research.
Large language models (LLMs) are increasingly used as synthetic survey respondents, but existing evaluations ask whether answers look plausible at the individual level. We argue the right question is psychometric: do LLMs preserve the joint distribution, latent structure, reliability, mediation pathways, and demographic effects of real human survey data? We introduce a Lithuanian organisational-psychology dataset (n=263 employees; Dunham Attitudes Toward Change, UWES-17, Koopmans IWPQ; 68 items, 12 subscales) and condition a 37-model lineup spanning OpenAI, Anthropic, Google, and twelve open-weight families on real respondent profiles under a five-level persona-disclosure ladder, presentation and reasoning-effort ablations, counterfactual demographic swaps (gender, role, education), a cross-language check, and a verbatim-recall memorization probe. The resulting Psychometric Similarity Score (PSS) is anchored against five non-LLM statistical baselines and a held-out human-vs-human ceiling, with respondent-bootstrap confidence intervals and an item-permutation null for Tucker's phi. LLMs reproduce the qualitative direction of human psychometric relationships, but a Gaussian-copula baseline beats every LLM on the sample-driven PSS components; the LLM"crowd"is more similar to itself (mean inter-LLM PSS 0.73) than to humans; and memorization does not drive the leaderboard (recall-PSS rank correlation 0.00). Counterfactual swaps reveal education-driven effects (mean |d|=0.56) that dwarf gender (0.12) and role (0.18); Tucker's phi on UWES falls inside the permutation null for 8 of 37 models. Downstream, every LLM shows a strong acquiescence shift (+0.84 SD), synthetic-trained regressors lose predictive validity on held-out humans (mean R^2 -0.18 vs 0.28), and models fabricate indirect effects on 3 of 10 placebo mediation paths. LLM samples are not a drop-in replacement for human survey data.