Search papers, labs, and topics across Lattice.
The paper introduces Continual Learning Bench (CL-Bench), a novel benchmark designed to evaluate the continual learning capabilities of LLM-based systems across six diverse real-world domains. By employing expert-validated tasks that share a learnable latent structure, CL-Bench effectively isolates the ability of stateful systems to learn from sequential experiences compared to stateless ones. The evaluation reveals that current models, including dedicated memory systems, often struggle with knowledge reuse and overfitting, highlighting significant room for improvement in continual learning methodologies.
Current LLMs often overfit and fail to leverage past knowledge, with naive in-context learning outperforming dedicated memory systems in continual learning tasks.
Continual learning, the ability of AI systems to improve through sequential experience, has attracted substantial interest, but no high-quality benchmark exists to evaluate it. We introduce Continual Learning Bench (CL-Bench), the first difficult, expert-validated benchmark designed to measure whether LLM-based systems genuinely improve with experience. CL-Bench spans six diverse domains (software engineering, signal processing, disease outbreak forecasting, database querying, strategic game-playing, and demand forecasting), each validated by domain experts and designed so that tasks share a learnable latent structure (codebase layout, disease outbreak dynamics, opponent strategies) that a stateful system can discover online but a stateless one cannot. We evaluate frontier models across several agent architectures, from naive in-context learning (ICL) to dedicated memory systems, introducing a gain metric to isolate learning from prior capabilities. We find that these systems leave headroom for improved continual learning: agents frequently overfit to immediate observations or fail to reuse knowledge across instances, and dedicated memory systems do not fix this -- in fact, naive ICL outperforms systems dedicated to memory management. CL-Bench is the first benchmark to evaluate continual learning across diverse real-world domains with expert-validated tasks and isolate online learning from underlying model capability, showing a need for better continual learning systems.
BAIR