Search papers, labs, and topics across Lattice.
This paper addresses the limitations of traditional Text-based Image Retrieval (TBIR) systems that rely on a single-match assumption by introducing a Domain-Specific Multi-Match Text-based Image Retrieval (DSMM-TBIR) framework tailored for specific domains like surveillance. The authors construct a new benchmark, Security Multi-Match TBIR (SecMM-TBIR), featuring 50,000 surveillance images and 200 queries, and propose the Semantic-Aware Fine-Tuning (SAFT) framework to mitigate the issue of false negatives in contrastive learning. Experimental results show that SAFT significantly improves retrieval performance, achieving an average mAP@20 gain of 7.8 points over standard methods while also enhancing general-domain capabilities.
Traditional TBIR methods fall short in domains like surveillance, but a new benchmark and fine-tuning approach can dramatically enhance retrieval accuracy.
Driven by the rapid advancement of vision-language representation learning, Text-based Image Retrieval (TBIR) has made notable progress. However, existing benchmarks are predominantly constructed on an exclusive single-match assumption between query and images. While effective in general scenarios, this assumption fails to reflect practical system performance in specific domains (e.g., surveillance), where a single query often corresponds to multiple relevant candidate images. To address this limitation, we design a Domain-Specific Multi-Match Text-based Image Retrieval (DSMM-TBIR) data engine. Leveraging this engine, we construct Security Multi-Match TBIR (SecMM-TBIR), a benchmark comprising 50k surveillance images with 200 comprehensive queries. Furthermore, we observe that vanilla contrastive learning in specific domains suffers from severe false negatives, forcing the model to push apart semantically similar pairs and thus degrading retrieval performance. We propose the Semantic-Aware Fine-Tuning (SAFT) framework to address semantic compression in specific domains, which incorporates Semantic-Aware Soft-Label Supervision (SASS) and Intra-modal Structural Distillation (ISD) to establish a promising paradigm for domain-specific TBIR tasks. Experiments across diverse CLIP-like models demonstrate that SAFT yields an average mAP@20 gain of 7.8 points on SecMM-TBIR over standard image-text contrastive (ITC) fine-tuning, while also improving general-domain performance. The entire benchmark will be released to facilitate further research.
