Search papers, labs, and topics across Lattice.
This study evaluates the ability of instruction-tuned large language models (LLMs) to recognize Schwartz's ten basic values through a controlled top-1 recognition task using 1,000 Russian situational texts. The results reveal that while the models achieve a pooled accuracy of 68.3% at top-1 and 89.2% at top-3, they frequently misidentify adjacent values, with over half of the semantic errors arising from confusion between closely related values. Notably, the study identifies eight recurring directed confusions that vary in severity across different model checkpoints, highlighting the need for a more nuanced evaluation of value recognition in LLMs.
LLMs misidentify adjacent values over 50% of the time, revealing critical biases that could skew their understanding of human motivations.
Large language models are increasingly evaluated through the values they endorse, but such evaluations presuppose that models can identify the value expressed in a concrete situation. We study this prerequisite as controlled top-1 recognition over Schwartz's ten basic values. Our evaluation set contains 1,000 Russian situational texts, balanced across the ten values and independently labeled by two human annotators per item. We evaluate 21 instruction-tuned LLM runs under a fixed ranked-response protocol; 20 runs with reliable outputs form the semantic panel. Pooled Acc@1 is 0.683 and Acc@3 is 0.892, showing that models often locate the correct motivational region while ranking close alternatives unstably. Adjacent values account for 50.9% of semantic errors, compared with 24.4% under a checkpoint-specific null. Eight directed confusions recur across checkpoints and human-confirmed subsets. Several are strongly asymmetric, including Universalism to Benevolence, Tradition to Conformity, and Security to Power, whereas Stimulation-Hedonism forms a bidirectional boundary. Their severity is checkpoint-specific and can bias higher-order value profiles. The results motivate value-recognition evaluation that combines exact accuracy, ranked recovery, and directed error analysis.
