Search papers, labs, and topics across Lattice.
This paper introduces TF-Engram, a train-free memory system that enhances Large Language Models (LLMs) by integrating offline constructed, phrase-specific semantic memory stored across a GPU-DRAM-SSD hierarchy. By employing Early-Exit Guided Predictive Prefetching, TF-Engram effectively mitigates latency during autoregressive decoding, leading to improved performance on downstream tasks. The results show a significant increase in average downstream scores, highlighting the system's ability to expand knowledge without the need for costly retraining or fine-tuning.
TF-Engram achieves a notable performance boost in LLMs by integrating scalable, train-free semantic memory without the typical overhead of retraining.
Large Language Models (LLMs) store factual knowledge and domain-specific patterns implicitly in dense Transformer parameters, making knowledge expansion costly through pretraining, fine-tuning, retrieval augmentation, or longer contexts. Engram-style memory offers a compact hidden-state injection pathway, but existing GPU-resident designs often rely on hash-based compression, causing unrelated phrases to collide in shared slots and weakening phrase-level semantic fidelity. We present TF-Engram, a train-free Engram system that constructs phrase-specific semantic memory offline from external corpora, stores large memory tables across a GPU--DRAM--SSD hierarchy, and uses Early-Exit Guided Predictive Prefetching to hide external-memory latency during autoregressive decoding. On Qwen3-0.6B, TF-Engram improves the average downstream score from 57.6 to 59.4, outperforming both the frozen backbone and a parameter-matched LoRA baseline. System evaluation shows that large TF-Engram tables can be built with moderate offline cost, SSD-backed storage substantially reduces GPU memory demand, and predictive prefetching recovers much of the throughput loss caused by external memory access. These results demonstrate that static phrase memory can be integrated into LLM inference as a scalable, train-free, and low-overhead system component.
