Search papers, labs, and topics across Lattice.
This paper introduces Doc-REFRAG, a novel framework designed to enhance multimodal document retrieval-augmented generation (RAG) by addressing the inefficiencies of existing models that struggle with multi-image scenarios. By utilizing a large-scale dataset of 343K question-answer pairs and implementing a question-guided approach that compresses and selectively expands visual tokens, the authors achieve significant improvements in accuracy and inference latency. The results demonstrate that Doc-REFRAG surpasses eleven strong baselines, establishing a new state-of-the-art in multimodal RAG performance.
Achieving state-of-the-art accuracy in multimodal RAG while drastically reducing inference latency could redefine efficiency standards in document-based question answering.
Real-world knowledge resides in multimodal documents, necessitating retrieval-augmented generation (RAG) for accurate question answering. However, existing multimodal RAG models are primarily designed for single-image or closed-document settings and exhibit limited accuracy in realistic multi-image scenarios. Moreover, processing numerous retrieved images incurs substantial computational overhead from irrelevant visual tokens. To address these challenges, we introduce DocLongRAG, a large-scale dataset of 343K question--answer pairs, each associated with an average of 37.4 retrieved images to reflect authentic RAG workflows. Building on this dataset, we propose Doc-REFRAG, a question-guided framework that compresses visual tokens into coarse chunks and selectively expands question-relevant ones via a lightweight RL-based selector. Experiments on six benchmarks show that Doc-REFRAG outperforms eleven strong baselines, achieving state-of-the-art accuracy with significantly lower inference latency. Our resources are available at https://github.com/Collab-Gen/Doc-REFRAG.
