Search papers, labs, and topics across Lattice.
This paper identifies critical flaws in existing Composed Image Retrieval (CIR) benchmarks due to indeterminate queries and lack of multi-round evaluation. To address this, the authors introduce FISD, a new benchmark with precisely controlled image pairs generated using generative models, and an automatic multi-round agentic evaluation framework. Experiments using FISD and the multi-round framework demonstrate a more realistic and nuanced evaluation of CIR models, revealing limitations not apparent in previous benchmarks.
Current Composed Image Retrieval benchmarks are misleading, as a new evaluation reveals that models struggle with query ambiguity and interactive scenarios.
Composed Image Retrieval (CIR) aims to retrieve a target image based on a query composed of a reference image, and a relative caption that specifies the desired modification. Despite the rapid development of CIR models, their performance is not well characterized by existing benchmarks, which inherently contain indeterminate queries degrading the evaluation (i.e., multiple candidate images, rather than solely the target image, meet the query criteria), and have not considered their effectiveness in the context of the multi-round system. Motivated by this, we consider improving the evaluation procedure from two aspects: 1) we introduce FISD, a Fully-Informed Semantically-Diverse benchmark, which employs generative models to precisely control the variables of reference-target image pairs, enabling a more accurate evaluation of CIR methods across six dimensions, without query ambiguity; 2) we propose an automatic multi-round agentic evaluation framework to probe the potential of the existing models in the interactive scenarios. By observing how models adapt and refine their choices over successive rounds of queries, this framework provides a more realistic appraisal of their efficacy in practical applications. Extensive experiments and comparisons prove the value of our novel evaluation on typical CIR methods.
