Search papers, labs, and topics across Lattice.
This paper introduces PeFuse, a training-free framework for Composed Image Retrieval (CIR) that utilizes pretrained Diffusion Models and Multimodal Large Language Models to facilitate modality bridging through generative conversion. By reformulating CIR into single-modality retrieval tasks via uni-directional and bi-directional conversion strategies, the method eliminates the need for task-specific training while achieving competitive performance on standard benchmarks. The findings reveal that this pseudo-fusion approach not only enhances retrieval effectiveness but also offers significant flexibility through its modular design.
Transforming Composed Image Retrieval into a training-free framework could redefine how we approach multimodal search tasks.
Composed Image Retrieval (CIR) is an emerging paradigm in content-based image retrieval that enables users to formulate compositional queries by combining a reference image with an auxiliary modality, usually text-based. This approach supports fine-grained search where the target image shares structural elements with the user-provided image while incorporating the modifications specified by the auxiliary text. Conventional CIR methods rely on multimodal fusion to combine visual and textual features into a joint query embedding, which requires training modules that align composed queries with the targets. In this work, we propose PeFuse (for pseudo-fusion), a training-free framework that leverages pretrained Diffusion Models and Multimodal Large Language Models to bridge modalities via generative conversion. We introduce two novel strategies: uni-directional and bi-directional conversion, which convert CIR into four single-modality retrieval problems. These methods reformulate CIR as either intra-modal or cross-modal single-query retrieval tasks, bypassing the need for dedicated task-specific training. Extensive experiments on standard benchmarks demonstrate that converting CIR into text-to-image retrieval tasks is more effective than alternative conversion strategies, achieving competitive or superior performance compared with state-of-the-art methods, while maintaining high flexibility thanks to replaceable components of the conversion pipeline. These results highlight the effectiveness of the pseudo-fusion paradigm for zero-shot CIR. Our code is publicly available at: https://github.com/StevenXuf/PeFuse4CIR.
