Search papers, labs, and topics across Lattice.
This paper investigates the efficacy of precomputed memory in language models, specifically focusing on how it maintains correctness and the conditions that lead to its degradation. Through experiments with Llama-3.1-8B-Instruct, the authors find that while precomputed memories can reduce the need for repeated context feeding, they require costly rebuilds to stay current and often fail to incorporate corrections unless specifically phrased. The results suggest that optimizing the rebuilding process and serving updates alongside existing memories can significantly enhance the performance of language models in dynamic query environments.
Precomputed memory in language models can degrade significantly unless rebuilt frequently and updated with specifically phrased corrections.
Language models can answer from precomputed memory, a model's saved reading of a body of material, reused across requests instead of read again at each. This paper maps where that practice preserves correctness and the conditions under which it fails. Across experiments on Llama-3.1-8B-Instruct using both saved key-value caches and trained compressions of them, precomputed memory degrades when assembled from separately prepared parts, stays current only through rebuilds costing a large fraction of full preparation in our measurements, and ignores corrections served beside it conditional on phrasing. If precomputed memories can be served alongside one another, be cost-efficiently rebuilt, and be superseded by new information arriving in real-time, they can serve as a way to avoid re-feeding context to a model over repeated queries. The implication of our results for a deployed system that deals with a variety of queries is that precomputed memories are best rebuilt on the cadence at which new information changes what the memory was originally computed from. Both warm-rebuilding trained compressions of key-value caches and serving specifically-phrased updates beside a memory, as pasted text or injected cache state, show particular promise for keeping precomputed memories current, the latter as an interim measure between rebuilds, and we measure the cost and name the remaining questions associated with each.
