Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study explores the annotation of social media texts for human values using Schwartz's theoretical framework, focusing on how various LLMs and prompts influence value expression. By employing iterative prompt calibration and error analysis, the authors demonstrate that different LLMs yield distinct interpretations of values, and they enhance annotation accuracy through targeted expert verification rules. The findings reveal that LLM-generated annotations can be effectively transferred to an encoder model, preserving theoretical consistency and uncertainty information in value expression.
Different LLMs interpret human values in social media texts variably, but targeted calibration can significantly enhance alignment with expert judgments.
Measuring subjective constructs in naturally occurring social media text requires annotation procedures that are theoretically grounded, empirically validated, and transferable to an encoder model for scalable prediction. Using non-English social media posts annotated according to Schwartz's theory of basic human values, we investigate how different LLMs, prompts, and instruction languages operationalize the expression of values in text. We argue that although texts may permit multiple plausible interpretations, theory-based value definitions can constrain interpretations and reduce spurious value attributions. Beyond precision, recall, and F1, we evaluate structural alignment between values, error structure, confidence-ambiguity relations, and annotation stability. We show that different LLMs produce different value interpretations. Iterative prompt calibration through error analysis reduces misattributions and improves alignment with expert annotations. We also derive targeted expert verification rules from recurrent error structures and use them during corpus annotation. Finally, we show that LLM annotations can be transferred to an encoder model through soft-label training, retaining theory-based value interpretations and information about uncertainty in value expression.