Search papers, labs, and topics across Lattice.
This paper introduces QIMG-7, a benchmark designed to evaluate the impact of polluted multimodal retrieval on multi-sentence factual question answering (QA). The authors demonstrate that naive multimodal fusion significantly deteriorates performance when faced with unreliable content, with a drop from 0.908 to 0.490 in the gpt-4o-mini stack. By implementing a source-aware trust resolution (SATR) approach, they achieve a balanced score of 0.816, highlighting the importance of selective trust over unconditional fusion in multimodal retrieval scenarios.
Naive multimodal fusion fails dramatically under polluted conditions, revealing that selective trust can significantly enhance QA performance.
Multimodal retrieval-augmented generation (RAG) is often evaluated with clean evidence, yet real retrieval can return topically relevant but unreliable content: false text and misleading images from corrupted metadata, entity swaps, typographic overlays, semantic edits, adversarial patches, blends, or style transfer. We introduce QIMG-7, a controlled benchmark for multimodal retrieval pollution in multi-sentence factual QA, spanning four datasets, seven image-attack families, and 16 paired clean/polluted regimes, for 1,760 evaluation rows per method. Across four generator/gate stacks, naive multimodal fusion is brittle: in the main gpt-4o-mini stack, Full-MM support drops from 0.908 with clean text to 0.490 with polluted text, often making Parametric fallback safer than retrieval. We propose source-aware trust resolution (SATR), a training-free approach that compares Parametric, Text-only, and Full-MM candidate answers and selects among candidate answers or falls back based on source reliability. The Field-Selector variant achieves the best balanced score, 0.816, improving over Full-MM by 11.7 points and over the Cascaded Router by 2.7 points. Ablations show that, in this text-first setting, explicit text-reliability modeling is the dominant driver of these gains. Overall, in text-first factual QA with multimodal retrieval conflict, our results support selective trust rather than unconditional fusion. Artifacts are available at https://github.com/SaadElDine/Trust_Before_Fusion.
