Search papers, labs, and topics across Lattice.
This paper introduces the continuous-query limited memory language model (CO-LMLM), which enhances the externalization of factual knowledge by using continuous keys paired with textual knowledge values in a knowledge base. This novel approach allows for flexible vector queries that integrate human-readable knowledge into text generation, significantly improving knowledge control compared to traditional language models. Experimental results show that CO-LMLM surpasses previous limited memory models and standard LLMs in perplexity and factual precision, achieving competitive performance with much larger models while using less data.
CO-LMLM achieves lower perplexity than models trained on 40 times more data, revolutionizing how we leverage knowledge bases in language generation.
Limited memory language models (LMLMs) externalize factual knowledge during pretraining to a knowledge base (KB), rather than memorizing it in their weights. During generation, the model then fetches knowledge from the KB as needed. This recently introduced paradigm provides multiple advantages, including knowledge control capabilities that remain beyond conventional LLMs. We propose continuous-query LMLM (CO-LMLM), where the KB pairs continuous keys with textual knowledge values, a significant departure from prior reliance on relational KB and queries. CO-LMLM generates flexible vector queries at minimal cost, while still integrating human-readable and attributable retrieved knowledge into its generation. We pair this design with an annotation pipeline that tags free-form factual spans in arbitrary text, removing prior work's restriction to Wikipedia. Across pretraining on Wikipedia and FineWeb-Edu and at multiple model scales, CO-LMLM outperforms prior LMLMs and vanilla LLMs in both perplexity and factual precision. At 360M scale, this includes lower perplexity than models pretrained on 40x more data, and SimpleQA-verified performance that is in line with gpt-4o-mini and higher than Claude Sonnet 4.5.
