Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study compares the annotation capabilities of trained linguists, linguists in training, and large language models (LLMs) in the context of evaluative language within TED talk transcripts, focusing on the Appraisal theory's Attitude subsystem. The research reveals that while trained linguists struggle with agreement on subjective annotations, LLMs outperform them, achieving an F1-score of 0.77 after fine-tuning. These findings suggest that LLMs can effectively assist in complex annotation tasks, potentially transforming approaches in digital humanities research.
LLMs outshine trained linguists in annotating subjective evaluative language, achieving higher accuracy in complex linguistic tasks.
In this paper, we draw a comparison between linguists in training, a trained linguist, and annotations generated by large language models (LLMs) to find out if they struggle with complex linguistic phenomena in a similar way. For this purpose, we analyse evaluative language in spoken popular science discourse, with the example of a corpus of English TED talk transcripts. We focus on the Appraisal theory and its Attitude subsystem, including the categories (classes) of Affect, Judgement, and Appreciation. In this context, Appraisal theory is an example of a highly subjective annotation task, making it a suitable example for the study of complex annotation challenges. First, we assess human annotations on a sentence level in specific scientific domains. Then, we develop three prompts and compare them for model performance for the automatic classification of Appraisal classes. We assess the performance of three LLMs using the best-performing prompt and finetune the model, reaching an F1-score of 0.77. We find that models perform best compared to annotations conducted by the trained linguist, while linguists in training do not reach high agreement scores. We conclude that LLMs can aid in complex annotation task resolution, opening new pathways for the complex theories annotated and analyzed in digital humanities studies.