Search papers, labs, and topics across Lattice.
This paper explores using Vision-Language Models (VLMs) as Collaborative Filtering (CF)-aware multimodal encoders for Sequential Recommendation (SR). They find that standard contrastive supervised fine-tuning (SFT) of VLMs leads to modality collapse, where one modality dominates optimization while the other degrades. To mitigate this, they propose VLM2Rec, which introduces Weak-modality Penalized Contrastive Learning and Cross-Modal Relational Topology Regularization to ensure balanced modality utilization, achieving state-of-the-art results.
Naive fine-tuning of VLMs for multimodal sequential recommendation causes catastrophic modality collapse, but can be fixed with gradient rebalancing and cross-modal regularization.
Sequential Recommendation (SR) in multimodal settings typically relies on small frozen pretrained encoders, which limits semantic capacity and prevents Collaborative Filtering (CF) signals from being fully integrated into item representations. Inspired by the recent success of Large Language Models (LLMs) as high-capacity embedders, we investigate the use of Vision-Language Models (VLMs) as CF-aware multimodal encoders for SR. However, we find that standard contrastive supervised fine-tuning (SFT), which adapts VLMs for embedding generation and injects CF signals, can amplify its inherent modality collapse. In this state, optimization is dominated by a single modality while the other degrades, ultimately undermining recommendation accuracy. To address this, we propose VLM2Rec, a VLM embedder-based framework for multimodal sequential recommendation designed to ensure balanced modality utilization. Specifically, we introduce Weak-modality Penalized Contrastive Learning to rectify gradient imbalance during optimization and Cross-Modal Relational Topology Regularization to preserve geometric consistency between modalities. Extensive experiments demonstrate that VLM2Rec consistently outperforms state-of-the-art baselines in both accuracy and robustness across diverse scenarios.
