Search papers, labs, and topics across Lattice.
The paper introduces a benchmark for evaluating episodic memory capabilities in LLMs, motivated by the need to improve LLM reasoning and reduce confabulation by grounding them in real-world events. The benchmark includes a structured representation of episodic events with temporal, spatial, and entity information, and is designed to be free of contamination. Evaluation of state-of-the-art models (GPT-4, Claude, Llama 3.1, o1-mini) on the benchmark reveals significant deficiencies in episodic memory, especially with multiple related events or complex spatio-temporal relationships.
LLMs still struggle to remember even short sequences of events, revealing a critical gap between current capabilities and human-like cognition.
Episodic memory -- the ability to recall specific events grounded in time and space -- is a cornerstone of human cognition, enabling not only coherent storytelling, but also planning and decision-making. Despite their remarkable capabilities, Large Language Models (LLMs) lack a robust mechanism for episodic memory: we argue that integrating episodic memory capabilities into LLM is essential for advancing AI towards human-like cognition, increasing their potential to reason consistently and ground their output in real-world episodic events, hence avoiding confabulations. To address this challenge, we introduce a comprehensive framework to model and evaluate LLM episodic memory capabilities. Drawing inspiration from cognitive science, we develop a structured approach to represent episodic events, encapsulating temporal and spatial contexts, involved entities, and detailed descriptions. We synthesize a unique episodic memory benchmark, free from contamination, and release open source code and datasets to assess LLM performance across various recall and episodic reasoning tasks. Our evaluation of state-of-the-art models, including GPT-4 and Claude variants, Llama 3.1, and o1-mini, reveals that even the most advanced LLMs struggle with episodic memory tasks, particularly when dealing with multiple related events or complex spatio-temporal relationships -- even in contexts as short as 10k-100k tokens.
