Search papers, labs, and topics across Lattice.
This paper benchmarks the serving costs of three memory systems (Mem0, Hindsight, and Mastra Observational Memory) against traditional strategies for long-running conversational agents. The study reveals that serving costs cannot be accurately predicted based solely on conversation length and message size, with internal memory behavior playing a significant role. Additionally, the analysis shows that the cost-effectiveness of memory systems varies greatly depending on the specific system and backbone used, with no single system outperforming others in both accuracy and cost efficiency across all scenarios tested.
Serving costs of memory systems can deviate by up to 69% from predictions based on conversation length, revealing hidden complexities in agentic memory performance.
Long-running conversational agents increasingly rely on a memory system to avoid resending the whole conversation each turn, yet how much that costs to serve has received little systematic benchmarking. We compare three memory systems (Mem0, Hindsight, and Mastra Observational Memory) against two reference strategies -- a fixed-size rolling window and resubmitting the full transcript -- across two backbones and conversations of up to 400 turns, pairing every cost measurement with answer accuracy on 665 LoCoMo questions. First, a memory system's serving cost cannot be predicted from conversation length and message size alone: a regression that tracks the two reference strategies closely misses the memory systems by 18-69%, their cost driven instead by internal memory behavior. Second, a break-even analysis shows that whether -- and when -- a memory system becomes cheaper to serve than the full transcript is highly sensitive to the system and the backbone, from the first tens of turns for the cheapest to never within 400 turns for the most expensive. Third, no system wins on both axes: accuracy spans 21-54%, and the backbone choice drives cost as much as the memory system does.
