Search papers, labs, and topics across Lattice.
The paper identifies two key failure modes in universal multimodal retrieval (UMR) models: visual modality collapse in early-fusion models like Marvel, and semantic misalignment in late-fusion models like UniVL-DR. To address these, they propose MiMIC, a fusion-in-decoder architecture trained with single modality mixin and random caption dropout. Experiments on WebQA+ and EVQA+ demonstrate that MiMIC outperforms both early- and late-fusion baselines, particularly in scenarios where images lack captions.
Early fusion UMR models lean too heavily on text, while late fusion struggles to relate semantically similar content – MiMIC offers a fix.
Universal Multimodal Retrieval (UMR) aims to map different modalities (e.g., visual and textual) into a shared embedding space for multi-modal retrieval. Existing UMR methods can be broadly divided into two categories: early-fusion approaches, such as Marvel, which projects visual features into the language model (LM) space for integrating with text modality, and late-fusion approaches, such as UniVL-DR, which encode visual and textual inputs using separate encoders and obtain fused embeddings through addition. Our pilot study reveals that Marvel exhibits visual modality collapse, which is characterized by the model's tendency to disregard visual features while depending excessively on textual cues. In contrast, although UniVL-DR is less affected by this issue, it is more susceptible to semantic misalignment, where semantically related content is positioned far apart in the embedding space. To address these challenges, we propose MiMIC, which introduces two key innovations: (1) a fusion-in-decoder architecture for effective multimodal integration, and (2) robust training through single modality mixin and random caption dropout. Experiments on the WebQA+ and EVQA+ datasets, where image in documents or queries might lack captions, indicate that MiMIC consistently outperforms both early- and late-fusion baselines.
