Search papers, labs, and topics across Lattice.
This paper introduces ProMSA, a progressive multimodal search agent designed for Knowledge-based Visual Question Answering (KB-VQA), which enhances the traditional fixed retrieve-then-generate pipeline by allowing adaptive tool selection during reasoning. By employing a rejection-sampling supervised fine-tuning (SFT) approach for training and optimizing with a sequence-level reinforcement learning objective (TN-GSPO), ProMSA effectively balances tool-use formats and interaction depth. Experimental results on E-VQA and InfoSeek demonstrate significant improvements in retrieval and end-to-end accuracy compared to established retrieval-augmented generation (RAG) and agent baselines.
ProMSA achieves superior accuracy in KB-VQA by dynamically selecting retrieval strategies, outperforming traditional fixed pipelines.
Knowledge-based Visual Question Answering (KB-VQA) requires models to combine image understanding with external knowledge. Most prior methods use a fixed retrieve-then-generate pipeline with a pre-selected retriever and a static top-k setting, which is not adaptive during reasoning. We propose ProMSA, a progressive multimodal search agent for KB-VQA. Given an image-question pair, the agent iteratively chooses image search, text search, or stop, under explicit tool-call budgets and with deduplication to avoid redundant retrieval. For training, we first use rejection-sampling SFT to learn valid tool-use formats, then optimize the agent with TN-GSPO, a sequence-level RL objective that normalizes updates by both generation length and tool-interaction depth. Experiments on E-VQA and InfoSeek show consistent gains over strong RAG and agent baselines, and improved retrieval and end-to-end accuracy. The code is available at https://github.com/DingWu1021/Promsa.
Tsinghua AI