Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
WikiSeeker is a multi-modal retrieval-augmented generation (RAG) framework for Knowledge-Based Visual Question Answering (KB-VQA) that uses VLMs for query refinement and retrieval inspection. It introduces a Refiner agent that rewrites the textual query based on the input image to improve retrieval and an Inspector agent that routes reliable retrieved context to an LLM or relies on the VLM's internal knowledge when retrieval is unreliable. Experiments on EVQA, InfoSeek, and M2KR show WikiSeeker achieves state-of-the-art performance in retrieval accuracy and answer quality by more effectively leveraging VLMs.
Stop using VLMs just for answer generation: WikiSeeker shows how to unlock their potential in KB-VQA by using them as specialized agents for query refinement and retrieval inspection.
Multi-modal Retrieval-Augmented Generation (RAG) has emerged as a highly effective paradigm for Knowledge-Based Visual Question Answering (KB-VQA). Despite recent advancements, prevailing methods still primarily depend on images as the retrieval key, and often overlook or misplace the role of Vision-Language Models (VLMs), thereby failing to leverage their potential fully. In this paper, we introduce WikiSeeker, a novel multi-modal RAG framework that bridges these gaps by proposing a multi-modal retriever and redefining the role of VLMs. Rather than serving merely as answer generators, we assign VLMs two specialized agents: a Refiner and an Inspector. The Refiner utilizes the capability of VLMs to rewrite the textual query according to the input image, significantly improving the performance of the multimodal retriever. The Inspector facilitates a decoupled generation strategy by selectively routing reliable retrieved context to another LLM for answer generation, while relying on the VLM's internal knowledge when retrieval is unreliable. Extensive experiments on EVQA, InfoSeek, and M2KR demonstrate that WikiSeeker achieves state-of-the-art performance, with substantial improvements in both retrieval accuracy and answer quality. Our code will be released on https://github.com/zhuyjan/WikiSeeker.