Search papers, labs, and topics across Lattice.
This paper introduces a two-stage generator-in-the-loop alignment framework designed to enhance the performance of vision-language models (VLMs) in retrieval-augmented generation (RAG) tasks by bridging the semantic-utility gap. The first stage generates a hypothetical passage from an image-query pair to inform retrieval, while the second stage fine-tunes a cross-encoder reranker using answer-supervised preference pairs derived from a frozen VLM. Experimental results on VQA-X and A-OKVQA demonstrate that this approach significantly outperforms traditional methods, highlighting the effectiveness of answer-level feedback in improving document relevance for generating correct answers.
Answer-level feedback from VLMs can dramatically improve document retrieval effectiveness in multimodal tasks, outperforming traditional semantic similarity metrics.
Vision-language models (VLMs) augmented with retrieval-augmented generation (RAG) benefit from access to external evidence. However, standard retrievers and rerankers optimize for semantic similarity rather than answer utility, creating a preference gap: documents that appear relevant may not help the generator produce a correct answer. Motivated by this, we propose a two-stage generator-in-the-loop alignment framework that closes this gap without human document-level relevance annotations. Our framework consists of two stages: in Stage 1, a VLM generates a hypothetical text passage from the image-query pair, which is used as the retrieval query for dense text search, bridging the image-to-text modality gap. In Stage 2, a cross-encoder reranker adapted with low-rank adaptation (LoRA) is fine-tuned using answer-supervised preference pairs mined from the frozen VLM: given the dataset answer label, a candidate document is labeled positive if the VLM produces the correct answer when given that document as context, and negative otherwise. This generator-guided signal is compatible with multiple alignment loss functions, including contrastive (triplet) loss, pairwise direct preference optimization (DPO), and supervised fine-tuning (SFT), and supports periodic re-mining to refresh preference pairs as the reranker improves. Experiments on VQA-X and A-OKVQA with Qwen3.5-2B and Qwen3-VL-4B-Instruct show that our proposed framework consistently outperforms rank-order, random, and REPLUG-style likelihood baselines under various alignment losses and pool size settings, suggesting that answer-level generator feedback is an effective supervision signal for preference alignment.
