Search papers, labs, and topics across Lattice.
This paper introduces Contextual Composed Image Retrieval (CoCo-IR), a novel task designed to enhance image retrieval systems by allowing users to iteratively refine their search results across multiple interactions. The authors propose a Large Multimodal Model (LMM) that acts as a context-aware reasoner, generating Transformable Image Embeddings (TIE) that adapt based on the entire interaction history. Experimental results show that CoCo-IR achieves state-of-the-art performance, with a mAP@5 of 39.4 on the single-turn benchmark CIRCO and a remarkable 44.1 R@1 on the new CoCo-IR benchmark for 4-turn dialogues, significantly outperforming existing methods.
Iterative image retrieval can now adapt contextually, achieving a 44.1 R@1 on multi-turn searches鈥攆ar surpassing previous benchmarks.
Current instruction-based image retrieval systems are powerful but limited to single-turn interactions, failing to capture the iterative nature of complex, real-world visual searches. To overcome this limitation, we introduce Contextual Composed Image Retrieval (CoCo-IR), a novel task that enables users to progressively refine search results through interactions. We address this new task by proposing a new model based on a Large Multimodal Model (LMM) that functions as a context-aware reasoner for CoCo-IR. Our model interprets the entire interaction history to generate Transformable Image Embeddings (TIE) that evolve across turns. To fuel the model training without expensive human annotations, we develop a fully autonomous, scalable data engine that leverages LMMs to generate high-quality contextual retrieval data, and uses model-guided verification to mine challenging hard negatives. Extensive experiments demonstrate that our approach establishes new state-of-the-art performance: We achieve 39.4 mAP@5 on the challenging single-turn benchmark CIRCO; furthermore, on our new CoCo-IR benchmark, our model maintains robust performance with 44.1 R@1 on 4-turn dialogues, dramatically outperforming existing methods (28.2 4-turn R@1) that fail to handle multi-turn context. Project page: https://CoCo-IR.github.io.
