Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the ability of large language models (LLMs) to evaluate social attraction based on theory-grounded persona profiles derived from psychological constructs. Through three studies involving 34 LLMs and 198 human participants, the researchers found that LLMs exhibited strong stability and consistency in their ratings, aligning with human judgments in structure but diverging in intensity. Notably, while both LLMs and humans maintained a three-tier ordering of profiles, LLMs were more extreme in their evaluations, rating attractive profiles higher and unattractive profiles lower than humans did.
LLMs can reliably assess social attraction but tend to exaggerate their judgments compared to human evaluators.
Large language models (LLMs) are increasingly used to perform subjective evaluations traditionally made by humans, yet their validity as social judges remains unclear. This paper examines whether LLMs can assess social attraction from theory-grounded persona profiles constructed from ten psychological and relational constructs and organized into three tiers: socially attractive, socially mixed, and socially unattractive. We examine LLM ratings in two studies and compare them with human judgments in a third study. In Study 1, 34 LLMs rated 12 profiles across three repeated runs. Although some models tended to give higher or lower ratings overall, they showed strong stability across runs, consistent three-tier ordering, and high agreement in relative profile ordering. Study 2 examined sensitivity to gender presentation using six matched name-and-pronoun profile pairs and a separate pronoun-only test with a gender-neutral name, finding no significant effects in either analysis. In Study 3, 198 human participants evaluated the six matched profiles from Study 2. Their ratings reproduced the three-tier structure and followed a profile ordering consistent with that of the LLMs. However, LLMs rated attractive profiles more positively and unattractive profiles more negatively than humans, while neither group showed a significant overall effect of gender presentation.