Search papers, labs, and topics across Lattice.
This paper analyzes memory compaction strategies in large language models (LLMs) and agents through a rate-distortion framework, unifying disparate approaches to memory management. By establishing a comprehensive taxonomy and a single compaction objective, the authors reveal that attention magnitude and recency consistently dictate what information is retained or discarded across various layers. Key findings highlight that current methods often prematurely discard critical information before queries are known, leading to inefficiencies that could be addressed through new benchmarks and design principles.
Memory compaction in LLMs is fundamentally flawed, with critical information often discarded before it's needed, revealing a systemic inefficiency across all layers.
Large language models, and the agents built on them, spend an ever-growing share of their compute and memory on remembering: caching attention keys and values, carrying long prompts, maintaining recurrent state, and storing what happened in previous turns and sessions. Because none of this memory is free, four largely separate research communities have each learned to compact it. They evict or quantize the KV cache, prune or distill prompts, bound architectural state, and consolidate agent memory. We argue that these are instances of one problem: a rate--distortion decision about what context-derived information to retain versus discard, at what fidelity, under a resource budget, so as to preserve downstream task utility. We make this lens precise with a single compaction objective and a layer-agnostic lower bound, use it to build a seven-axis taxonomy that classifies methods from across the stack uniformly, and use it to transfer mechanisms between layers that have never been connected, from serving-stack KV management to agent long-term memory. Two patterns hold across the survey. At every layer the signal that decides what to keep is attention magnitude or recency, and it fails in the same way everywhere, by discarding, before the query is known and with no way to undo it, information the query later needs. And while compression is measured carefully on single-turn long context, the repeated compaction that agents actually perform is almost never measured, and no benchmark holds one budget axis across all the layers at once. We turn both observations into a benchmark proposal, a small reference experiment, and a set of compaction-aware design principles, and we map the open problems.
