Search papers, labs, and topics across Lattice.
This study introduces ArcANE, a novel benchmark designed to evaluate role-playing language agents (RPLAs) based on their ability to maintain character consistency throughout evolving narratives. By segmenting character arcs into psychological phases and testing models on both in-text and out-of-text scenarios, the authors demonstrate that conditioning on these arcs significantly enhances performance, particularly in novel situations. The results show that fine-tuning models with ArcANE data leads to even greater improvements, highlighting the importance of character evolution in narrative-based AI interactions.
Conditioning on character arcs boosts RPLA performance by over 40% in novel scenarios, revealing the critical role of psychological evolution in storytelling.
Role-playing language agents (RPLAs) should play characters whose values and behavior evolve as the story progresses, not maintain a fixed persona. Existing benchmarks measure factual recall at a given chapter, not whether responses align with the character's psychological trajectory, especially in scenarios the source text never explores. We introduce ArcANE (Arc-Aware Narrative Evaluation), an automatically constructed benchmark spanning 17 novels and 80 principal characters. A Character Arc segments the narrative into phases along a psychological axis, and each probe poses the same scenario across phases, spanning both situations within the source text and situations beyond it. Across six models and six context modes, conditioning on the Character Arc tops every other context strategy on every model, and the gap is largest on scenarios outside the source text where retrieval has nothing to find. We further fine-tune open-weight models on the same data to obtain ArcANE-8B/32B, which widen the Arc advantage even more on scenarios outside the source text.
