Search papers, labs, and topics across Lattice.
This paper introduces MULVEC, a role-aware method for training-free zero-shot composed image retrieval that enhances the matching process by structuring queries into four distinct roles: Global, Desired, Preserve, and Forbidden. By utilizing frozen encoders to generate a target description vector alongside role-specific probe vectors, MULVEC effectively retains fine-grained details that traditional global matching methods often overlook. The approach demonstrates significant improvements, achieving up to a 23.0% increase in mAP@5 on the CIRCO dataset compared to the best existing methods, while also outperforming benchmarks on CIRR and FashionIQ.
Role-aware matching in image retrieval can boost performance by up to 23% without the need for task-specific training.
Training-free zero-shot composed image retrieval finds a target image in a gallery from a reference image and a text edit without learning from task-specific image triplets. Existing methods typically describe the target as a whole and match this description with a global image representation. This global matching can mix different semantic cues and lose fine- grained details. We propose MULVEC, a role-aware method whose compiler produces a structured query record that is mapped to four retrieval roles: Global describes the full target, Desired states what should appear, Preserve states what should remain, and Forbidden states what should disappear. Frozen encoders map the query to one target description vector and role-specific probe vectors, while each candidate is represented by one global visual vector and a bank of local visual vectors. The retrieval roles then use this shared evidence for their respective purposes, and a fixed weighted sum of their scores ranks the entire gallery in a single retrieval pass. Across CIRCO, CIRR, and FashionIQ and three backbone scales, MULVEC improves CIRCO mAP@5 by up to 23.0% over the strongest compared method and gives the best CIRR and FashionIQ results in our comparison.
