Search papers, labs, and topics across Lattice.
This paper analyzes a large dataset of graded health-literacy annotations of COVID-19 responses, finding that disagreement among annotators is more strongly driven by the conceptual difficulty of the question than by individual annotator biases. By stratifying the analysis based on inter-annotator agreement, the authors demonstrate that key social-scientific effects exhibit significant variation and even reversals across different agreement levels. This highlights the importance of perspectivist modeling approaches that account for structured disagreement to avoid obscuring crucial inferential differences in graded interpretive tasks.
Discarding disagreement as noise in health-literacy annotation can mask critical social-scientific effects, which only surface when analyzing data stratified by inter-annotator agreement.
Annotation pipelines in Natural Language Processing (NLP) commonly assume a single latent ground truth per instance and resolve disagreement through label aggregation. Perspectivist approaches challenge this view by treating disagreement as potentially informative rather than erroneous. We present a large-scale analysis of graded health-literacy annotations from 6,323 open-ended COVID-19 responses collected in Ecuador and Peru. Each response was independently labeled by multiple annotators using proportional correctness scores, reflecting the degree to which responses align with normative public-health guidelines, allowing us to analyze the full distribution of judgments rather than aggregated labels. Variance decomposition shows that question-level conceptual difficulty accounts for substantially more variance than annotator identity, indicating that disagreement is structured by the task itself rather than driven by individual raters. Agreement-stratified analyses further reveal that key social-scientific effects, including country, education, and urban-rural differences, vary in magnitude and in some cases reverse direction across levels of inter-annotator agreement. These findings suggest that graded health-literacy evaluation contains both epistemically stable and unstable components, and that aggregating across them can obscure important inferential differences. We therefore argue that strong perspectivist modeling is not only conceptually justified but statistically necessary for valid inference in graded interpretive tasks.
