Search papers, labs, and topics across Lattice.
This paper introduces CMDR, a novel framework for contextual multimodal document retrieval that jointly encodes multiple pages to capture document context, addressing limitations in existing benchmarks that rely on independent page evaluations. The authors also present CMCL, a contrastive learning objective that enhances the training of CMDR-Embed by balancing the need for contextual understanding with the discriminability of individual pages. Experimental results show that CMDR-Embed outperforms traditional non-contextual embeddings, underscoring the critical role of context in multimodal retrieval tasks.
Context-aware embeddings can drastically improve multimodal document retrieval, revealing insights that independent page evaluations miss.
Multimodal document retrieval aims to retrieve relevant pages while preserving both textual and visual content from the original document. However, existing benchmarks primarily evaluate simple lexical or semantic matching, and most methods encode pages independently. Consequently, they overlook the contextual information in the document required to resolve queries that aggregate information across multiple pages. In this paper, we introduce CMDR and CMDR-Bench, a new multimodal document retrieval task and benchmark that require modeling document context. To address this challenge, we propose CMDR-Embed, a contextual multimodal embedding framework that explicitly incorporates document context by jointly encoding multiple pages and deriving page-level embeddings from a shared contextual representation. Furthermore, we introduce CMCL, a contextual multimodal contrastive learning objective that effectively trains CMDR-Embed by balancing contextual modeling with page-level discriminability. Experiments demonstrate that CMDR-Embed significantly outperforms non-contextual embeddings, highlighting the importance of context-aware multimodal embeddings for advancing document retrieval.
