Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces DREAM, a novel method that leverages the autoregressive next-token prediction objective of large language models (LLMs) to train dense retrieval embeddings without the need for costly labeled document pairs. By integrating query-document similarity scores into specific attention heads of a frozen LLM, DREAM effectively aligns the LLM's predictive capabilities with the retrieval task, allowing the model to generate gradients for retriever training. The evaluation on BEIR and RTEB benchmarks shows that DREAM consistently surpasses existing baselines across various model sizes, highlighting its potential as a robust training strategy for dense retrievers.
Autoregressive modeling can transform dense retrieval training by eliminating the need for expensive labeled data, achieving superior performance on retrieval benchmarks.
Dense retrieval embedding models are a fundamental component of modern retrieval-based AI systems. Most dense retrievers are trained with contrastive objectives, which require labeled positive and negative document pairs that are often costly and difficult to obtain. In this work, we investigate whether the autoregressive next-token prediction objective of a large language model (LLM) can provide supervision for dense retrieval. The intuition is simple: if a document contains information relevant to a query, conditioning on that document should make the target output easier for the LLM to predict. A key challenge is that the next-token prediction loss is computed inside the LLM, while the retriever is a separate embedding model. To address this challenge, we propose DREAM (Dense Retrieval Embeddings via Autoregressive Modeling), which injects retriever-generated query-document similarity scores into selected attention heads of a frozen LLM. During training, these scores determine how much attention each candidate document receives while the LLM predicts the target output. The resulting prediction loss provides gradients for retriever training through the attention mechanism. We evaluate DREAM on retrieval benchmarks BEIR and RTEB using embedding backbones ranging from 0.5B to 3B parameters. DREAM consistently outperforms existing baselines across different model scales. These results demonstrate that DREAM provides a promising approach for training dense retrievers through autoregressive modeling.