Search papers, labs, and topics across Lattice.
This paper introduces REMORY, a neural memory network that augments textual summaries with a fixed-size sequence of learned soft memory tokens to enable long-horizon agents to operate within finite context windows. REMORY learns to generate these tokens, conditioned on the summary, to allow a frozen LLM to approximate its full-history output, effectively acting as a residual connection along the sequence dimension. Experiments show REMORY achieves near full-context performance on source attribution and joint scores with significantly reduced input positions, while also improving tool usage consistency in long-horizon agent benchmarks.
A residual memory mechanism allows LLMs to retain crucial context for long-horizon tasks, achieving near full-context performance with only 5.2% of input positions.
Long-horizon agents compact their history to continue within a finite context window, but a textual summary alone may not support every subsequent decision. We introduce REMORY, a neural memory network that supplements the summary with a bounded sequence of soft memory tokens. Given the history and summary, the network learns to generate tokens that help a frozen LLM approximate the continuation it would produce with the full history. The tokens are conditioned on the summary and appended after it, forming an analogue of a residual connection along the sequence dimension. On SummHay, REMORY improves source attribution at nearly unchanged insight coverage and approaches the full-context joint score using only 5.2% of the input positions. Across long-horizon agent benchmarks, Qwen3.8-27B and GLM-5.3-Flash show consistent gains with residual memory. Both models also exhibit substantially fewer repeated tool outputs and tool errors on BrowseComp and Terminal-Bench 2.1.
