Search papers, labs, and topics across Lattice.
This paper introduces two new corpora, Hlava Cor and Hlava AD, designed to investigate human label variation in coreference and discourse relations within Czech texts. By analyzing annotations from multiple annotators, the study reveals significant discrepancies in how individuals interpret text coherence, with inter-annotator agreement hovering around 60-65%. Notably, lower agreement in coreference annotations correlates with instances where automatic models also struggle, highlighting the complexities of human interpretation in challenging contexts.
Human annotators exhibit significant variability in coreference and discourse relation judgments, revealing insights into the subjective nature of text coherence that automatic models often miss.
As previous research on annotator disagreement in discourse phenomena has shown, understanding text coherence varies considerably from one individual to another. To explore this phenomenon, we created two corpora with multiple annotations of Czech texts, accompanied by annotators' explanations of their choices. The first corpus consists of 1,024 contexts annotated in parallel by three annotators. It captures differences in the identification of coreference across various text types and grammatical-semantic categories, including pronouns, full noun phrases, and anaphoric adverbials. The second corpus comprises 512 contexts, annotated in parallel by five annotators, and focuses on identifying discourse relations in attributive and non-attributive constructions. Both corpora achieve a comparable inter-annotator agreement of approximately 60-65%. For coreference annotation, agreement tends to be lower in cases where automatic coreference resolution models disagree, suggesting that when the models disagree, the examples tend to be more difficult or ambiguous for human annotators to interpret. The annotators' comments, both for coreference and discourse relations, further reveal differences in interpretation, varying levels of confidence in text understanding, and individual reading strategies.
