Search papers, labs, and topics across Lattice.
This paper introduces SCALE-QA, a novel benchmark designed to evaluate conversational memory in chat assistants by focusing on flat, mixed-topic threads that require the system to infer relevant prior episodes for task completion. The benchmark comprises 3,000 questions across various domains, challenging existing long-context models and retrieval-augmented generation (RAG) systems. The proposed Temporal-Semantic Interleaved Memory Reconstruction (TSIM) method significantly enhances memory accuracy, outperforming strong baselines by 5.6 to 17.6 percentage points across multiple LLM backends.
Forgetting previous episodes in mixed-topic conversations can lead to significant accuracy drops, but TSIM shows how to maintain context integrity and improve performance in chat assistants.
Conversations with chat assistants increasingly span many topics in a single long-running thread, challenging memory systems. Existing long-context and memory benchmarks often expose session or topic boundaries, or probe direct personal-memory questions. These settings understate a harder assistant-memory regime: a flat mixed-topic thread where the system must infer which earlier episode makes a later task decision valid. We introduce SCALE-QA, a constraint-grounded task QA benchmark for flat unsegmented threads targeting episode integrity failure. The dataset contains 3,000 audited questions across 10 domains, uses deterministic four-way multiple-choice grading, and includes a deterministic runtime builder; experiments use all 3,000 questions through 128k and a stratified 400-question diagnostic at 1M. SCALE-QA questions are ordinary task-oriented requests whose correct answer depends on causally related evidence introduced earlier in the conversation. We also propose Temporal-Semantic Interleaved Memory Reconstruction (TSIM), which segments the turn stream into coherent episodes and indexes them through a hierarchical multi-view memory stack with deterministic episode-level summary and cluster-routing views. Experiments show that SCALE-QA challenges strong RAG baselines and long-context LLMs alike; across three open-source and proprietary LLM backends, TSIM achieves the highest accuracy in every backend setting, gaining 5.6-17.6 accuracy points over the strongest corresponding baseline.
