Search papers, labs, and topics across Lattice.
This study conducts a comprehensive analysis of narrative features within the Dolma corpus, a 3-trillion-token dataset used for LLM pretraining, by applying a framework based on narrative theory that focuses on agency, setting, and events. By fine-tuning NarraBERT, a RoBERTa-based model, the authors validate their framework through the annotation of 400 passages and extend their findings to a larger dataset of 3 million passages, resulting in the creation of NarraDolma. The results reveal measurable narrative structures across diverse data sources and highlight significant disparities in narrative qualities that current data curation practices overlook, thereby offering insights into how these factors influence narrative reasoning in LLMs.
Narrative structures in web-scale LLM training data are not only measurable but also reveal significant disparities that current curation practices fail to address.
The narrative composition of web-scale LLM pretraining corpora remains largely unexplored even though narrative is a fundamental mode of human communication. We present the first fine-grained study of narrative features in Dolma, a 3-trillion-token open pretraining corpus. Drawing on narrative theory, we design a framework spanning three core narrative elements (agency, setting, and events) operationalized as 11 interpretable dimensions. After sampling and annotating a diverse set of 400 passages, we finetune and validate NarraBERT, a RoBERTa-based model for fine-grained narrative prediction. We apply NarraBERT to 3M passages, resulting in a new dataset, NarraDolma. We find (i) narrative structure is measurable at scale across extremely heterogeneous data, (ii) we uncover a continuous, multidimensional narrative structure underlying web text, and (iii) narrative qualities are unequally distributed across pretraining sources and topics in ways that current curation practices neither measure nor account for. Our framework, dataset, and analyses provide a foundation for understanding how narrative qualities are distributed in LLM pretraining data and for studying how data composition affects narrative reasoning tasks. We publicly release NarraDolma and NarraBERT.
ETH