Search papers, labs, and topics across Lattice.
This paper investigates how well LLMs integrate information across long-form texts by comparing LLM-generated novel summaries to human-written summaries. They align sentences from 150 human summaries with specific novel chapters to measure "conceptual engagement" and compare this to LLM-generated summaries. The key finding is that LLMs disproportionately emphasize the ends of novels compared to humans, suggesting a degradation in narrative comprehension across long contexts.
LLMs focus on the ends of novels when summarizing, unlike humans, revealing a potential bottleneck in long-context narrative understanding.
Although LLM context lengths have grown, there is evidence that their ability to integrate information across long-form texts has not kept pace. We evaluate one such understanding task: generating summaries of novels. When human authors of summaries compress a story, they reveal what they consider narratively important. Therefore, by comparing human and LLM-authored summaries, we can assess whether models mirror human patterns of conceptual engagement with texts. To measure conceptual engagement, we align sentences from 150 human-written novel summaries with the specific chapters they reference. We demonstrate the difficulty of this alignment task, which indicates the complexity of summarization as a task. We then generate and align additional summaries by nine state-of-the-art LLMs for each of the 150 reference texts. Comparing the human and model-authored summaries, we find both stylistic differences between the texts and differences in how humans and LLMs distribute their focus throughout a narrative, with models emphasizing the ends of texts. Comparing human narrative engagement with model attention mechanisms suggests explanations for degraded narrative comprehension and targets for future development. We release our dataset to support future research.
