Search papers, labs, and topics across Lattice.
This paper introduces HMGCLIP, a novel multimodal embedding framework that enhances product representation learning by addressing the limitations of existing Multimodal Large Language Models (MLLMs) in capturing fine-grained product attributes. By utilizing a heterogeneous hypergraph structure, HMGCLIP effectively mines hard negatives and aligns multi-granular semantics, allowing for improved discrimination of subtle differences in product attributes. Experimental results demonstrate that HMGCLIP significantly outperforms leading multimodal encoders and e-commerce baselines on both a new fine-grained dataset and the MAVE benchmark, underscoring its effectiveness in e-commerce applications.
HMGCLIP achieves superior attribute discrimination in e-commerce by leveraging a novel hypergraph structure that captures both fine and coarse-grained product semantics.
Although recent Multimodal Large Language Models (MLLMs) have advanced general product understanding, they implicitly encode product information into global embeddings, thereby limiting their ability to capture fine-grained attributes. This limitation hinders performance in tasks requiring precise attribute discrimination, such as distinguishing subtle material differences among visually similar products. To address this challenge, we propose HMGCLIP, a unified multimodal embedding framework. By constructing a heterogeneous hypergraph, we leverage hypergraph topology to mine structure-aware hard negatives and align multi-granular semantics at both relation and hyperedge levels. This design enables a dual-granularity inference mechanism that dynamically fuses attribute evidence for both fine-grained and coarse-grained downstream tasks. Furthermore, we release a comprehensive fine-grained e-commerce dataset to facilitate future benchmarking. Extensive experiments on this new dataset and the public MAVE benchmark show that HMGCLIP outperforms strong multimodal encoders, MLLMs, and e-commerce baselines, validating the superiority of HMGCLIP.
DAMO