Search papers, labs, and topics across Lattice.
This study enhances Multimodal Large Language Models (MLLMs) for complex image retrieval tasks by introducing a novel two-stage fine-tuning strategy and an automated dataset construction pipeline for fine-grained multimodal quintuple datasets. The approach separates the fine-tuning process into context reasoning and retrieval-oriented stages, significantly improving the model's ability to understand context and align queries with targets. Experimental results across five diverse datasets reveal that this method outperforms existing techniques in zero-shot retrieval scenarios, even with a lighter MLLM backbone.
Fine-grained context modeling boosts MLLM retrieval performance, achieving superior results in complex tasks with a lightweight architecture.
Due to their strong generalizable multimodal processing and reasoning capabilities, Multimodal Large Language Models (MLLMs) have demonstrated significant potential as universal image retrievers, effectively addressing diverse real-world image retrieval tasks. Nevertheless, pioneering studies, while promising, overlook the potential of fine-grained context modeling and disentangled fine-tuning objectives in enhancing MLLMs' retrieval performance, particularly for complex tasks such as long-text-to-image retrieval, visual dialog retrieval, and composed image retrieval (CIR). Therefore, in this work, we propose an automated fine-grained multimodal quintuple dataset construction pipeline and a novel two-stage fine-grained multimodal fine-tuning strategy. The dataset generation pipeline produces a comprehensive CIR dataset with fine-grained image captions and modification text, facilitating fine-grained context modeling. Beyond the previously entangled fine-tuning paradigm, our approach separates the fine-tuning process into two distinct stages: (1) fine-grained context reasoning-oriented fine-tuning and (2) fine-grained retrieval-oriented fine-tuning. These stages aim to sequentially enhance the model's context understanding and query-target alignment capabilities, thereby improving retrieval performance. Extensive experiments across five datasets encompassing diverse and complex image retrieval tasks demonstrate the remarkable superiority of our method over existing approaches in zero-shot retrieval settings, even with a more lightweight MLLM backbone compared to those methods.
