Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This survey analyzes the evolution of contextual memory mechanisms in LLMs, moving beyond dense self-attention's quadratic prefill cost and growing KV cache. It introduces a five-dimensional lens (Representation, Update, Access, Readout, Integration) to compare diverse approaches like explicit memory, sparse access, and recurrent states, revealing their convergence in controlling memory functions. The analysis of 59 model records shows increasing depth coordination and a multidimensional memory-routing hypothesis, indicating efficient sequence architecture design now centers on memory organization and selective use.
LLM memory evolution reveals a shift from isolated attention to coordinated, multidimensional memory routing across depth and substrate, challenging the focus on single operators.
Self-attention gives LLMs fine-grained, query-dependent access to context, but dense token interactions incur quadratic prefill cost and a key--value cache growing with context length. Research thus spans explicit-memory compression, sparse access, recurrent state construction, structured state dynamics, and heterogeneous mechanism composition. This survey analyzes these developments as model-internal contextual memory. We introduce a five-dimensional lens---Memory Representation, Memory Update, Access, Readout, and Integration---describing what is represented, how it changes, what is query-eligible, how it is read, and how readouts form outputs. This lens compares overlapping research lines without imposing one computational model. We reconstruct mechanism-level developments and architectural adoption using 59 release-level records from 14 major model lineages and 11 high-performing open-weight endpoints. First, explicit-memory and recurrent-state methods retain distinct interfaces but increasingly control overlapping memory functions. Second, heterogeneous architectures increasingly coordinate across network depth: layer-wise composition distributes complementary memory processing across representational stages, while cross-layer reuse carries selected memory and routing artifacts forward. Depth thus becomes a dimension along which contextual memory is constructed and managed. Third, these developments motivate a stateful multidimensional memory-routing hypothesis: persistent memory is organized across temporal scope, network depth, substrate type, and representation granularity, while coordinated Sparse Write and Sparse Read determine what is maintained and what contributes to each query. Overall, efficient sequence architecture design increasingly concerns the organization, lifecycle, and selective use of contextual memory rather than an isolated Attention operator.