Search papers, labs, and topics across Lattice.
The paper introduces Region-R1, a query-side region cropping framework for multi-modal re-ranking in retrieval-augmented generation, designed to mitigate the impact of visual distractors. Region-R1 employs a novel region-aware group relative policy optimization (r-GRPO) to dynamically crop a discriminative region from the query image before re-ranking retrieved candidates. Experiments on E-VQA and InfoSeek benchmarks demonstrate that Region-R1 achieves state-of-the-art performance, improving conditional Recall@1 by up to 20%.
By learning to focus on question-relevant image regions, Region-R1 boosts multi-modal re-ranking performance by up to 20%, showing that attending to the right visual cues is more important than seeing everything.
Multi-modal retrieval-augmented generation (MM-RAG) relies heavily on re-rankers to surface the most relevant evidence for image-question queries. However, standard re-rankers typically process the full query image as a global embedding, making them susceptible to visual distractors (e.g., background clutter) that skew similarity scores. We propose Region-R1, a query-side region cropping framework that formulates region selection as a decision-making problem during re-ranking, allowing the system to learn to retain the full image or focus only on a question-relevant region before scoring the retrieved candidates. Region-R1 learns a policy with a novel region-aware group relative policy optimization (r-GRPO) to dynamically crop a discriminative region. Across two challenging benchmarks, E-VQA and InfoSeek, Region-R1 delivers consistent gains, achieving state-of-the-art performances by increasing conditional Recall@1 by up to 20%. These results show the great promise of query-side adaptation as a simple but effective way to strengthen MM-RAG re-ranking.
