Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To disentangle permanent information loss from transient retrieval failure in agent memory systems, the authors implement a restore-counterfactual audit on LongMemEval-S that re-injects gold evidence into read-time contexts across FIFO, random, redundancy-aware, and LLM-importance eviction policies. Current budget-accuracy benchmarks obscure whether agent errors stem from retriever deficits or true eviction, masking whether sophisticated memory management actually outperforms naive baselines. The audit reveals that irreversible information loss hits 100% across all policies at tight budgets (8k tokens), while at 80k tokens, LLM-importance eviction only marginally edges out FIFO (0.60 vs. 0.67–0.73 irreversible error share), showing no detectable retention advantage in matched-accuracy comparisons.
Sophisticated agent memory eviction heuristics barely protect against permanent forgetting, with strict token budgets driving irreversible data loss to 100% regardless of whether LLM-importance pruning or naive FIFO is used.
Agent memory systems must discard stored information when their history exceeds a fixed token budget. Existing budget-accuracy frontiers quantify the resulting loss in accuracy, but do not distinguish irreversible losses caused by eviction from recoverable retrieval failures. We introduce the restore counterfactual, a per-question paired intervention that reinstates the question's gold evidence in the read-time context and reruns the same reader. Combining the change in correctness with whether the evidence was retained after eviction classifies each oracle-answerable error as recoverable, irreversible, or residual; in the residual case, the answer remains incorrect after restoration. We evaluate FIFO, random, redundancy-aware, and LLM-importance eviction on LongMemEval-S at three budgets and under two retrieval regimes, using GPT-4o-mini as the primary reader and judge and GPT-5.4-mini as a robustness reader. Under top-k retrieval at an 80k-token budget, the irreversible share among errors corrected by restoration is 0.67-0.73 for FIFO, random, and redundancy-aware eviction, compared with 0.60 for LLM-importance. At 8k tokens, it reaches 1.00 for all four policies. Recoverable errors occur under top-k retrieval at 80k tokens but are absent under forced-gold injection by construction, so budget-accuracy results are not directly comparable unless the retrieval regime is reported. An exploratory matched-accuracy analysis detects no difference in irreversible rate among accuracy-matched policy pairs at a resolution of 1.2-6 percentage points. The same analysis detects the deliberately destructive control. To our knowledge, this is the first per-item, per-question restore-counterfactual audit of eviction for external agent-memory stores on a standard conversational benchmark.