Search papers, labs, and topics across Lattice.
This paper investigates the suitability of annotator-specific and label distribution models for Implicit Discourse Relation Recognition (IDRR), a task known for high ambiguity. They find that annotator-specific models struggle in IDRR unless ambiguity is reduced, while models trained on label distributions offer more stable predictions. Their analysis reveals that cognitively demanding cases are a primary source of inconsistency in human interpretation, hindering the effectiveness of perspectivist modeling in IDRR.
Modeling annotator disagreement in discourse relation recognition is more effective with label distributions than with annotator-specific models, challenging the assumption that perspectivist models are universally superior for ambiguous NLP tasks.
There is growing recognition that many NLP tasks lack a single ground truth, as human judgments reflect diverse perspectives. To capture this variation, models have been developed to predict full annotation distributions rather than majority labels, while perspectivist models aim to reproduce the interpretations of individual annotators. In this work, we compare these approaches on Implicit Discourse Relation Recognition (IDRR), a highly ambiguous task where disagreement often arises from cognitive complexity rather than ideological bias. Our experiments show that existing annotator-specific models perform poorly in IDRR unless ambiguity is reduced, whereas models trained on label distributions yield more stable predictions. Further analysis indicates that frequent cognitively demanding cases drive inconsistency in human interpretation, posing challenges for perspectivist modeling in IDRR.
