Search papers, labs, and topics across Lattice.
This paper investigates whether LLM-generated labels can replace human labels in active learning for hostility detection, and if active learning is still necessary given the ability to label entire corpora with LLMs. Using a new dataset of German political TikTok comments, they compare seven annotation strategies across four encoders, finding that a classifier trained on LLM-generated labels achieves comparable F1-Macro to one trained on human annotations at a fraction of the cost. However, LLM-trained classifiers over-predict the positive class in ambiguous cases, highlighting the importance of considering error profiles beyond aggregate F1.
LLM-labeled data can match human-labeled data in aggregate performance for hostility detection, but be warned: the errors are systematically different, especially in nuanced cases.
Instruction-tuned LLMs can annotate thousands of instances from a short prompt at negligible cost. This raises two questions for active learning (AL): can LLM labels replace human labels within the AL loop, and does AL remain necessary when entire corpora can be labelled at once? We investigate both questions on a new dataset of 277,902 German political TikTok comments (25,974 LLM-labelled, 5,000 human-annotated), comparing seven annotation strategies across four encoders to detect anti-immigrant hostility. A classifier trained on 25,974 GPT-5.2 labels (\$43) achieves comparable F1-Macro to one trained on 3,800 human annotations (\$316). Active learning offers little advantage over random sampling in our pre-enriched pool and delivers lower F1 than full LLM annotation at the same cost. However, comparable aggregate F1 masks a systematic difference in error structure: LLM-trained classifiers over-predict the positive class relative to the human gold standard. This divergence concentrates in topically ambiguous discussions where the distinction between anti-immigrant hostility and policy critique is most subtle, suggesting that annotation strategy should be guided not by aggregate F1 alone but by the error profile acceptable for the target application.
