Search papers, labs, and topics across Lattice.
This paper introduces CRAG-MM-Diagnostics, a diagnostic benchmark for Knowledge-Intensive Visual Question Answering (KI-VQA) that provides stage-wise data annotations to dissect the performance of Vision-Language Models (VLMs) across various sub-problems. By evaluating both fully parametric and retrieval-augmented VLMs, the authors identify knowledge retrieval and reasoning as the primary bottleneck, while also revealing significant challenges in visual grounding and object identification. The study culminates in a proposed grounded bimodal RAG pipeline that enhances the accuracy of GPT-5 and Qwen by 13.3 and 8.5 percentage points, respectively, underscoring the need for stage-aware evaluation in KI-VQA systems.
Knowledge retrieval and reasoning are the main bottlenecks in KI-VQA, but a new diagnostic benchmark reveals deeper issues in visual grounding and object identification.
Knowledge-Intensive Visual Question Answering (KI-VQA) benchmarks evaluate Vision-Language Models (VLMs) as multimodal knowledge assistants by requiring external information beyond a provided image to answer questions. KI-VQA involves multiple sub-problems -referring expression understanding, visual grounding, object recognition, knowledge retrieval, and reasoning-yet existing benchmarks typically report only end-task accuracy, obscuring where failures arise. To analyze the full KI-VQA pipeline, we introduce CRAG-MM-Diagnostics, a diagnostic benchmark with stage-wise data annotations that isolate 1) language-based visual grounding, 2) object identification, and 3) knowledge retrieval and reasoning. We evaluate fully parametric and retrieval-augmented VLMs, providing fine-grained analyses using newly collected metadata, such as target ROIs, entity names, and visual complexity scores. Our results point to knowledge retrieval and reasoning as the primary bottleneck, but also highlight issues in the other parts of the KI-VQA pipeline, such as the fact that VLMs struggle with target object identification or that image retrievers struggle to integrate textual cues. These findings expose fundamental limitations in current KI-VQA systems and motivate stage-aware evaluation. We, lastly, leverage these findings to propose a grounded bimodal RAG pipeline that integrates a visual grounding module to crop targets before image retrieval, boosting GPT-5 and Qwen's respective accuracies by 13.3 and 8.5 percentage points.
Mila