Search papers, labs, and topics across Lattice.
This paper introduces PRECOG, a retrieval mechanism that leverages the unique properties of State-Space Models (SSMs) to reduce the prefill cost of retrieval-augmented generation from O(L_context) to O(1) per query. By pre-encoding document corpora as fixed-size SSM hidden states, PRECOG enables direct state injection at query time, effectively bypassing the need for in-context re-ingestion. The authors demonstrate a significant performance improvement with TENNs-LLM, achieving a 4500脳 speedup in prefill latency, making the system suitable for interactive applications on edge hardware.
PRECOG achieves a staggering 4500脳 reduction in prefill latency, transforming retrieval-augmented generation from sluggish to interactive on edge devices.
Retrieval-augmented generation (RAG) imposes a prefill cost proportional to retrieved context length, and -- with Transformer backbones -- a KV-cache that grows with each generated token. State-Space Models (SSMs) avoid the second cost by construction; we eliminate the first, collapsing prefill from $O(L_{context})$ to $O(1)$ per query. We introduce PRECOG (Pre-Computed Context Injection), a retrieval mechanism that exploits a property unique to SSMs: the fixed-size, position-agnostic recurrent hidden state is a complete summary of everything the model has read. PRECOG pre-encodes document corpora offline as SSM hidden states and injects the best-matching state directly at query time, bypassing in-context re-ingestion entirely. The same state-injection mechanism enables SMC (Structured Memory Consolidation): a hierarchical persistent memory with cognitive-domain clustering, an adjustable fidelity-vs-storage dial, and $O(1)$ session initialization, which consolidates short-term episodic states into long-term semantic memory and fuses both with retrieved corpus states at query time. We demonstrate the system on TENNs-LLM, a 1.2B-parameter gated-SSM language model with a 192 KB hidden state. PRECOG matches in-context RAG answer quality, reducing prefill latency from $\sim$27 s to $<$6 ms on edge hardware -- a $\sim$4500$\times$ speedup that crosses the threshold from unusable to interactive. The mechanism is architecturally impossible for Transformer KV-caches, which are position-entangled and grow linearly with context length.
