Search papers, labs, and topics across Lattice.
This paper investigates unified text-image fusion for two-tower retrieval models in e-commerce, addressing the underutilization of visual information in current systems. They find that domain-specific fine-tuning and a two-stage alignment strategy between query, product text, and images are crucial for effective multimodal retrieval. The authors introduce a novel modality fusion network that captures cross-modal complementary information, demonstrating its effectiveness on large-scale e-commerce datasets.
E-commerce retrieval gets a visual boost: domain-specific fine-tuning and two-stage alignment unlock the power of product images, outperforming text-only approaches.
Modern e-commerce search is inherently multimodal: customers make purchase decisions by jointly considering product text and visual informations. However, most industrial retrieval and ranking systems primarily rely on textual information, underutilizing the rich visual signals available in product images. In this work, we study unified text-image fusion for two-tower retrieval models in the e-commerce domain. We demonstrate that domain-specific fine-tuning and two stage alignment between query with product text and image modalities are both crucial for effective multimodal retrieval. Building on these insights, we propose a noval modality fusion network to fuse image and text information and capture cross-modal complementary information. Experiments on large-scale e-commerce datasets validate the effectiveness of the proposed approach.
