Search papers, labs, and topics across Lattice.
This paper evaluates various retrieval-augmented generation (RAG) pipelines, including traditional text-based and advanced multimodal approaches, to determine their effectiveness in handling documents with complex layouts and visual elements. The authors introduce a quantitative selection methodology that aids practitioners in choosing the most suitable RAG pipeline based on empirical performance metrics and resource limitations. Key findings reveal that while multimodal pipelines enhance retrieval quality, they also incur higher computational costs, necessitating a careful balance between performance and efficiency.
Multimodal RAG systems can significantly improve retrieval quality, but their computational demands may not justify the gains in all scenarios.
Retrieval-augmented generation (RAG) systems have traditionally relied on text-based pipelines that extract and retrieve information from documents. While efficient and lightweight, these approaches often struggle with documents where meaning is conveyed through layout, tables, and visual elements. Recent advances in multimodal pipelines, powered by vision-language models (VLMs), improve retrieval quality by jointly encoding visual and textual signals, but at increased computational and memory cost. We propose a quantitative, data-driven selection methodology that guides practitioners in choosing the most appropriate RAG pipeline for a given document corpus based on empirical effectiveness and resource constraints. We evaluate contemporary textual and multimodal pipelines, including dense and late-interaction architectures, analyze their trade-offs, and provide actionable guidance for balancing retrieval performance with system efficiency.
