Search papers, labs, and topics across Lattice.
This paper introduces a method for LLM agents to perform retrieval directly from their hidden states, bypassing the need for a separate embedding model. A lightweight projection head is trained to map LLM hidden states into an embedding space using a combination of alignment, contrastive, and rank distillation losses. Experiments on QReCC demonstrate that this approach retains 97% of baseline retrieval quality while simplifying the architecture and reducing latency.
Ditch the extra embedding model: LLMs can retrieve information almost as well using just their internal representations, cutting complexity and latency.
LLM agents that retrieve external knowledge typically generate a search query as text, then run a separate embedding model to encode it into a vector. This two-model pipeline adds infrastructure complexity and latency, yet is redundant: the LLM already encodes the full conversational context in its hidden states. We propose equipping LLM agents with native retrieval capability by adding a lightweight projection head that maps hidden states directly into the embedding space, eliminating the need for a separate embedding model. Trained with a combination of alignment, contrastive, and rank distillation losses, our method retains 97\% of baseline retrieval quality while enabling the LLM agent to search with its own representations. Experiments on the QReCC conversational search benchmark show competitive Recall@10 and MRR@10 compared to the standard generate-then-encode pipeline, with systematic ablations confirming the contribution of each loss component.
