Search papers, labs, and topics across Lattice.
This paper introduces a narrative graph annotation framework for economic news narratives, leveraging qualitative content analysis (QCA) to improve annotation quality. They constructed a dataset of inflation narratives represented as directed acyclic graphs (DAGs) and rigorously evaluated annotation quality using a factorial experiment varying narrative representation and distance metrics. The key finding is that locally-constrained representations combined with stricter distance metrics (non-overlap based) significantly reduce annotation variability, leading to more reliable narrative graphs.
Overlap-based metrics inflate reliability scores in narrative graph annotation, while locally-constrained graph representations can significantly reduce annotation variability.
Narratives in news discourse play a critical role in shaping public understanding of economic events, such as inflation. Annotating and evaluating these narratives in a structured manner remains a key challenge for Natural Language Processing (NLP). In this work, we introduce a narrative graph annotation framework that integrates principles from qualitative content analysis (QCA) to prioritize annotation quality by reducing annotation errors. We present a dataset of inflation narratives annotated as directed acyclic graphs (DAGs), where nodes represent events and edges encode causal relations. To evaluate annotation quality, we employed a $6\times3$ factorial experimental design to examine the effects of narrative representation (six levels) and distance metric type (three levels) on inter-annotator agreement (Krippendorrf's $\alpha$), capturing the presence of human label variation (HLV) in narrative interpretations. Our analysis shows that (1) lenient metrics (overlap-based distance) overestimate reliability, and (2) locally-constrained representations (e.g., one-hop neighbors) reduce annotation variability. Our annotation and implementation of graph-based Krippendorrf's $\alpha$ are open-sourced. The annotation framework and evaluation results provide practical guidance for NLP research on graph-based narrative annotation under HLV.
