Search papers, labs, and topics across Lattice.
This paper introduces Paracosm, a training-free zero-shot composed image retrieval (CIR) method that directly generates a "mental image" from a multimodal query using a Large Multimodal Model (LMM). To address the synthetic-to-real domain gap between the generated mental image and real database images, the method also generates synthetic counterparts for the real images, creating a synthetic "paracosm" for matching. Paracosm achieves state-of-the-art zero-shot CIR performance on four challenging benchmarks by matching the generated mental image with the synthetic counterparts of the database images.
Forget textual descriptions – this zero-shot image retrieval method hallucinates the target image directly, outperforming the state-of-the-art by creating a whole synthetic world to match against.
Composed Image Retrieval (CIR) is the task of retrieving a target image from a database using a multimodal query, which consists of a reference image and a modification text. The text specifies how to alter the reference image to form a ``mental image'', based on which CIR should find the target image in the database. The fundamental challenge of CIR is that this ``mental image''is not physically available and is only implicitly defined by the query. The contemporary literature pursues zero-shot methods and uses a Large Multimodal Model (LMM) to generate a textual description for a given multimodal query, and then employs a Vision-Language Model (VLM) for textual-visual matching to search the target image. In contrast, we address CIR from first principles by directly generating the ``mental image''for more accurate matching. Particularly, we prompt an LMM to generate a ``mental image''for a given multimodal query and propose to use this ``mental image''to search for the target image. As the ``mental image''has a synthetic-to-real domain gap with real images, we also generate a synthetic counterpart for each real image in the database to facilitate matching. In this sense, our method uses LMM to construct a ``paracosm'', where it matches the multimodal query and database images. Hence, we call this method Paracosm. Notably, Paracosm is a training-free zero-shot CIR method. It significantly outperforms existing zero-shot methods on four challenging benchmarks, achieving state-of-the-art performance for zero-shot CIR.
DeepMind