Search papers, labs, and topics across Lattice.
The paper introduces crOss-Modal sInkhorn maTching (OMIT), a novel network for image-text matching that leverages Optimal Transport and Cross-modal Mover's Distance to compute similarity between visual and textual fragments. OMIT incorporates Sinkhorn iterations for efficient approximation and integrates partial matching to reduce redundant alignments by focusing on local-to-global similarities. Experiments on Flickr30K and MS-COCO demonstrate that OMIT achieves superior performance in cross-modal matching, showcasing its effectiveness and focal matching capabilities.
Achieve state-of-the-art image-text matching by focusing on relevant fragments, using Optimal Transport to efficiently compute similarity between visual and textual elements.
Cross-modal matching, a fundamental task in bridging vision and language, has recently garnered substantial research interest. Despite the development of numerous methods aimed at quantifying the semantic relatedness between image-text pairs, these methods often fall short of achieving both outstanding performance and high efficiency. In this paper, we propose the crOss-Modal sInkhorn maTching (OMIT) network as an effective solution to effectively improving performance while maintaining efficiency. Rooted in the theoretical foundations of Optimal Transport, OMIT harnesses the capabilities of Cross-modal Mover's Distance to precisely compute the similarity between fine-grained visual and textual fragments, utilizing Sinkhorn iterations for efficient approximation. To further alleviate the issue of redundant alignments, we seamlessly integrate partial matching into OMIT, leveraging local-to-global similarities to eliminate the interference of irrelevant fragments. We conduct extensive evaluations of OMIT on two benchmark image-text retrieval datasets, namely Flickr30K and MS-COCO. The superior performance achieved by OMIT on both datasets unequivocally demonstrates its effectiveness in cross-modal matching. Furthermore, through comprehensive visualization analysis, we elucidate OMIT's inherent tendency towards focal matching, thereby shedding light on its efficacy. Our code is publicly available at https://github.com/ppanzx/OMIT.
