Search papers, labs, and topics across Lattice.
This paper analyzes a Setswana sentiment dataset to understand the decline in inter-annotator agreement (IAA) over time in long annotation campaigns. They find that temporal simultaneity, i.e., tweets labeled close in time, is the strongest predictor of IAA, with labels applied within one minute achieving significantly higher agreement. The study also benchmarks several multilingual encoders, including GPT-5 and Gemini, on the dataset, demonstrating substantial gains from fine-tuning.
Annotating data in temporal proximity is the single biggest factor in high-quality labels, dwarfing individual annotator speed or tweet content.
Annotation quality is difficult to sustain when campaigns span weeks or months with small annotator pools. We present a Setswana sentiment dataset of 3,565 tweets annotated by three native-speaker annotators across eight batches and examine why inter-annotator agreement (IAA) declines over time. Despite an aggregate Randolph's free-marginal Kappa of $κ= 0.76$, "excellent," per-batch $κ$ falls by more than 32 points across the annotation task. Through six targeted analyses, we find that (i) label confusion concentrates on the negative/neutral boundary, (ii) two annotators show run-length drift consistent with autopilot labeling, and (iii) the dominant predictor of $κ$ is temporal simultaneity: tweets labeled within one minute achieve $κ= 0.98$, while those labeled more than a day apart reach only $κ= 0.65$. Annotation speed and tweet-level linguistic features show no meaningful association with $κ$. We benchmark three open multilingual encoders and proprietary models (GPT-5 and Gemini) on three-class sentiment classification; fine-tuning yields gains of 29 to 43 macro-F1 points over pretrained baselines, with GPT-5 few-shot leading overall (62.2 macro-F1). We release the dataset, per-annotation timestamps, and analysis code to support reproducible quality auditing for future African language NLP resources.
