Search papers, labs, and topics across Lattice.
DocRetriever enhances multimodal document retrieval by combining layout-aware sparse embeddings with a reasoning-augmented, few-shot reranker. The layout-aware sparse embedding technique avoids OCR while capturing structural information, and the reranker uses demonstrations and optimized sampling for improved generalization. Evaluated on a new benchmark, MultiDocR, DocRetriever outperforms existing methods, demonstrating improved retrieval accuracy across diverse document types.
Forget OCR: DocRetriever achieves state-of-the-art multimodal document retrieval by cleverly combining layout-aware sparse embeddings with a reasoning-augmented reranker.
Multimodal documents contain diverse elements, such as tables, figures, and layouts, which can complicate retrieval tasks. While current approaches typically combine dense visual embedding models with supervised rerankers to achieve high-precision retrieval, they face inherent limitations. First, the coarse-grained nature of dense embeddings tends to obfuscate explicit semantics, failing to leverage structurally salient information. Second, supervised reranking models suffer from generalization bottlenecks, as their performance heavily relies on domain-specific training data. Furthermore, existing benchmarks often lack diverse assessment dimensions and comprehensive relevance annotations, limiting reliable evaluation. To address these challenges, we propose DocRetriever, a plug-and-play framework. It enhances visual retrieval via a layout-aware sparse embedding technique, enabling effective hybrid encoding without the overhead of optical character recognition (OCR). We also introduce a generalizable reranker that leverages reasoning-augmented demonstrations and optimized sampling to improve accuracy in few-shot settings. Finally, we construct a new benchmark, MultiDocR, to enable more rigorous evaluation. Experiments across diverse benchmarks validate DocRetriever's superiority over state-of-the-art methods.
