Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces AttentionRetriever, a novel retrieval model designed for long documents that addresses context-awareness, causal dependence, and scope of retrieval limitations in existing RAG systems. AttentionRetriever leverages attention mechanisms and entity-based retrieval to create context-aware embeddings for long documents and determine the relevant retrieval scope. Experiments demonstrate that AttentionRetriever significantly outperforms existing retrieval models on long document retrieval datasets while maintaining the efficiency of dense retrieval methods.
Attention layers, typically used for processing, can be repurposed as surprisingly effective and efficient long-document retrievers.
Retrieval augmented generation (RAG) has been widely adopted to help Large Language Models (LLMs) to process tasks involving long documents. However, existing retrieval models are not designed for long document retrieval and fail to address several key challenges of long document retrieval, including context-awareness, causal dependence, and scope of retrieval. In this paper, we proposed AttentionRetriever, a novel long document retrieval model that leverages attention mechanism and entity-based retrieval to build context-aware embeddings for long document and determine the scope of retrieval. With extensive experiments, we found AttentionRetriever is able to outperform existing retrieval models on long document retrieval datasets by a large margin while remaining as efficient as dense retrieval models.