Search papers, labs, and topics across Lattice.
This paper introduces the Multiplex Multimodal Representation Model (MMRM), which enhances e-commerce search ranking by leveraging diverse collaborative signals for fine-tuning Multimodal Large Language Models (MLLMs). By employing a unified framework that integrates task-specific tokens and projection layers, MMRM generates comprehensive multiplex item representations and task-specific user representations in a single inference pass. Experimental results show that MMRM significantly improves ranking efficiency and effectiveness, leading to substantial performance gains in the JD e-commerce search engine.
MMRM unlocks the full potential of multimodal representations in e-commerce search, achieving remarkable efficiency and effectiveness by integrating diverse collaborative signals.
Multimodal information is pivotal for e-commerce search ranking. Existing works leverage multimodal data typically by fine-tuning general Multimodal Large Language Models (MLLMs) via collaborative signals, subsequently integrating the derived representations into ranking models as item features. Despite their efficacy, these methods face two primary limitations: (1) they rely on a single collaborative signal for MLLM fine-tuning, failing to exploit the heterogeneous signals essential for multitask ranking; and (2) they treat multimodal representations as regular item features in ranking models, underutilizing their latent potential for user behavior modeling. To address these challenges, we propose the Multiplex Multimodal Representation Model (MMRM), a unified framework that aligns MLLMs with diverse collaborative signals. By employing a shared backbone with task-specific tokens and projection layers, MMRM simultaneously learns from multiple signals and generates comprehensive multiplex item representations in a single inference pass. Furthermore, we introduce a multiplex user representation strategy in ranking models, which derives task-specific user representations via search-based behavior sequence modeling leveraging multiplex item representations. Extensive experiments demonstrate MMRM's superior efficiency and effectiveness. Notably, MMRM has been successfully deployed in the JD e-commerce search engine, yielding significant performance gains for millions of daily users.
