Search papers, labs, and topics across Lattice.
This paper introduces SubtleMemory, a benchmark designed to evaluate fine-grained relational memory discrimination in long-horizon AI agents, addressing the inadequacies of existing benchmarks that fail to assess how agents manage complex memory relations. The benchmark features 1,522 instances across 10 long histories, incorporating nuanced relational contexts that challenge agents to recover and utilize relational structures effectively. Evaluation of various memory systems reveals significant weaknesses in current AI models regarding their ability to discriminate and leverage relational memory, highlighting the need for improved architectures in persistent AI assistants.
Current AI memory systems struggle with fine-grained relational discrimination, revealing critical gaps in their ability to manage complex memory interactions over time.
Persistent AI assistants, such as OpenClaw, accumulate large collections of related memories over long-term interactions. As these memories grow, they may reinforce one another, diverge across contexts, or directly conflict, making correct assistance depend on memory relations rather than isolated recall. Existing long-term memory benchmarks rarely probe how agents preserve and utilize such relations during downstream tasks. To address this gap, we introduce SubtleMemory, a benchmark for fine-grained relational memory discrimination in long-running AI agents. SubtleMemory constructs relation-controlled latent semantic artifacts whose variants instantiate complementary, nuanced, or contradictory relations, and embeds them into realistic user-agent histories, requiring agents to recover distributed relational structures during later queries and instructions. The benchmark contains 1,522 evaluation instances over 10 long histories, grounded in 1,090 relation-controlled memory-variant sets and spanning user-related and non-user-related queries. Evaluating six standalone memory systems, two Claw-style agents with native memory modules, and three Claw-style agents with plugin memory modules, we find that current systems remain weak on fine-grained relational memory discrimination. We further introduce diagnostic protocols that reveal distinct capability profiles across memory preservation, retrieval, and downstream reasoning stages.
