Search papers, labs, and topics across Lattice.
The paper introduces OAKS, a benchmark to evaluate LLMs' ability to adapt to continually evolving knowledge streams by tracking fine-grained fact changes across context chunks. OAKS includes two datasets, OAKS-BABI and OAKS-Novel, with dense annotations to measure the accuracy of change tracking. Experiments on 14 models reveal that current LLMs and agentic memory systems struggle with robust adaptation, exhibiting delays and susceptibility to distraction in streaming environments.
LLMs struggle to keep up with dynamically changing facts in continual knowledge streams, often exhibiting delays and getting distracted, despite advances in model size and agentic memory systems.
LLMs operating in dynamic real-world contexts often encounter knowledge that evolves continuously or emerges incrementally. To remain accurate and effective, models must adapt to newly arriving information on the fly. We introduce Online Adaptation to Continual Knowledge Streams(OAKS) to evaluate this capability, establishing a benchmark for online adaptation over streaming, continually updating knowledge. Specifically, the benchmark is structured as a sequence of fine-grained context chunks where facts change dynamically across time intervals. OAKS comprises two datasets: OAKS-BABI and OAKS-Novel, where individual facts evolve multiple times across context chunks. These datasets include dense annotations to measure whether models track changes accurately. Evaluating 14 models with varied inference approaches, we observe significant limitations in current methodologies. Both state-of-the-art models and agentic memory systems fail to adapt robustly on OAKS, demonstrating delays in state-tracking and susceptibility to distraction within streaming environments.
