Search papers, labs, and topics across Lattice.
This paper investigates whether improved retrieval or document representation drives performance gains in multilingual and visually rich retrieval-augmented generation (RAG). Through controlled experiments manipulating transcription and preprocessing methods while keeping the retrieval mechanism (BM25) constant, the authors find that better document representation is the key factor in closing benchmark gaps. The results challenge the assumption that end-to-end multimodal retrievers are inherently superior, highlighting the importance of high-quality document preparation.
Seemingly large gains from complex, learned retrieval models on multilingual and visual RAG benchmarks may simply be due to better document representation, as BM25 closes the gap when given the same high-quality input.
Retrieval-augmented generation (RAG) is a common way to ground language models in external documents and up-to-date information. Classical retrieval systems relied on lexical methods such as BM25, which rank documents by term overlap with corpus-level weighting. End-to-end multimodal retrievers trained on large query-document datasets claim substantial improvements over these approaches, especially for multilingual documents with complex visual layouts. We demonstrate that better document representation is the primary driver of benchmark improvements. By systematically varying transcription and preprocessing methods while holding the retrieval mechanism fixed, we demonstrate that BM25 can recover large gaps on multilingual and visual benchmarks. Our findings call for decomposed evaluation benchmarks that separately measure transcription and retrieval capabilities, enabling the field to correctly attribute progress and focus effort where it matters.
