Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces SQUARE, a training-free zero-shot composed image retrieval (ZS-CIR) framework that uses multimodal large language models (MLLMs) to improve retrieval accuracy. SQUARE employs Semantic Query-Augmented Fusion (SQAF) to enrich the query embedding with MLLM-generated captions, providing high-level semantic guidance. It also uses Efficient Batch Reranking (EBR), where an MLLM jointly reasons about top-ranked candidates presented as an image grid to refine the ranking in a single pass.
Forget finetuning: SQUARE uses MLLMs to boost zero-shot composed image retrieval, outperforming existing methods without any task-specific training.
Composed Image Retrieval (CIR) aims to retrieve target images that preserve the visual content of a reference image while incorporating user-specified textual modifications. Training-free zero-shot CIR (ZS-CIR) approaches, which require no task-specific training or labeled data, are highly desirable, yet accurately capturing user intent remains challenging. In this paper, we present SQUARE, a novel two-stage training-free framework that leverages Multimodal Large Language Models (MLLMs) to enhance ZS-CIR. In the Semantic Query-Augmented Fusion (SQAF) stage, we enrich the query embedding derived from a vision-language model (VLM) such as CLIP with MLLM-generated captions of the target image. These captions provide high-level semantic guidance, enabling the query to better capture the user's intent and improve global retrieval quality. In the Efficient Batch Reranking (EBR) stage, top-ranked candidates are presented as an image grid with visual marks to the MLLM, which performs joint visual-semantic reasoning across all candidates. Our reranking strategy operates in a single pass and yields more accurate rankings. Experiments show that SQUARE, with its simplicity and effectiveness, delivers strong performance on four standard CIR benchmarks. Notably, it maintains high performance even with lightweight pre-trained, demonstrating its potential applicability.