Search papers, labs, and topics across Lattice.
This paper introduces MMAgent-R$^2$, a novel framework for Knowledge-based Visual Question Answering (KB-VQA) that enhances retrieval accuracy by integrating visual reranking and active rejection mechanisms. By directly comparing query and candidate images, the model effectively distinguishes between visually similar entities, while active rejection allows for the retrieval of additional candidates when confidence is low, thereby mitigating the propagation of retrieval errors. Experimental results on multiple benchmarks show that MMAgent-R$^2$ outperforms existing methods, particularly in complex retrieval scenarios and multi-image reasoning tasks.
Visual reranking and active rejection in MMAgent-R$^2$ significantly boost retrieval accuracy in challenging KB-VQA tasks, outperforming traditional methods.
Knowledge-based Visual Question Answering (KB-VQA) requires models to retrieve visual entities matching the query image from large-scale encyclopedic knowledge bases and answer related questions. Existing multimodal Retrieval Augmented Generation (mRAG) methods rely on global visual features to match candidate entities, yet when the knowledge base contains numerous visually similar entities, the retriever struggles to distinguish them, populating the candidate set with visually similar but factually mismatched distractors. Since subsequent processing steps such as noise filtering are also confined to this fixed candidate set, errors from failed retrieval inevitably propagate to the final answer. To address these challenges, we propose MMAgent-R$^2$, an agentic mRAG framework that integrates visual reranking and active rejection as its internal verification mechanism. Visual reranking directly compares query and candidate images, capturing discriminative details beyond textual descriptions to precisely identify the target entity among similar candidates; active rejection discards unreliable results and retrieves additional candidates when no confident match is found, moving beyond the fixed candidate pool. We design a composite reward function with step-level verification rewards and achieve joint optimization of external retrieval, internal verification, and answer generation via GRPO training. Experiments on InfoSeek, E-VQA, and MMhops demonstrate that \ours{} achieves state-of-the-art performance, with particularly notable advantages in challenging retrieval scenarios and complex multi-image multi-hop reasoning tasks.
