Search papers, labs, and topics across Lattice.
The paper introduces Thought-Retriever, a model-agnostic algorithm that enables LLMs to leverage arbitrarily long external data by retrieving relevant intermediate responses ("thoughts") from past interactions, overcoming context length limitations. This approach filters redundant thoughts, organizes them into a thought memory, and retrieves relevant ones for new queries, effectively creating a self-evolving long-term memory for LLM agents. Experiments on a new AcademicEval benchmark and other datasets demonstrate that Thought-Retriever significantly outperforms state-of-the-art baselines, showing improvements in F1 score and win rate, and enabling LLMs to self-evolve and leverage deeper thoughts for abstract queries.
LLMs can now tap into arbitrarily long-term memories by retrieving "thoughts" – their own past reasoning steps – rather than just raw data, leading to significant performance gains.
Large language models (LLMs) have transformed AI research thanks to their powerful internal capabilities and knowledge. However, existing LLMs still fail to effectively incorporate the massive external knowledge when interacting with the world. Although retrieval-augmented LLMs are proposed to mitigate the issue, they are still fundamentally constrained by the context length of LLMs, as they can only retrieve top-K raw data chunks from the external knowledge base which often consists of millions of data chunks. Here we propose Thought-Retriever, a novel model-agnostic algorithm that helps LLMs generate output conditioned on arbitrarily long external data, without being constrained by the context length or number of retrieved data chunks. Our key insight is to let an LLM fully leverage its intermediate responses generated when solving past user queries (thoughts), filtering meaningless and redundant thoughts, organizing them in thought memory, and retrieving the relevant thoughts when addressing new queries. This effectively equips LLM-based agents with a self-evolving long-term memory that grows more capable through continuous interaction. Besides algorithmic innovation, we further meticulously prepare a novel benchmark, AcademicEval, which requires an LLM to faithfully leverage ultra-long context to answer queries based on real-world academic papers. Extensive experiments on AcademicEval and two other public datasets validate that Thought-Retriever remarkably outperforms state-of-the-art baselines, achieving an average increase of at least 7.6% in F1 score and 16% in win rate across various tasks. More importantly, we further demonstrate two exciting findings: (1) Thought-Retriever can indeed help LLM self-evolve after solving more user queries; (2) Thought-Retriever learns to leverage deeper thoughts to answer more abstract user queries.
CMU ML