Search papers, labs, and topics across Lattice.
This paper addresses the challenge of Incomplete Multimodal Sentiment Analysis with Unseen Modality Combinations (IMSAUMC), which arises when models encounter modality combinations during testing that were not present in training. The authors propose Contrastive Mixed Prompt Learning (CMPL), which employs a label-guided contrastive feature learning mechanism and modality-combination prompts to enhance model generalization. Experimental results show that CMPL outperforms state-of-the-art methods by over 5% in accuracy across three benchmark datasets, highlighting its effectiveness in real-world multimodal scenarios.
Models trained on incomplete multimodal data can significantly improve generalization to unseen combinations, achieving over 5% higher accuracy than existing approaches.
Incomplete multimodal sentiment analysis has garnered significant attention in recent years. Existing approaches typically assume that data is missing at random or are designed specifically for certain missing patterns, ignoring the modality combination inconsistency between training and testing phases. However, in real-world scenarios, the testing phase often encounters modal combinations that were not present during the training phase, which leads to insufficient generalization capabilities and unstable performance. In this paper, we introduce the problem of Incomplete Multimodal Sentiment Analysis with Unseen Modality Combinations (IMSAUMC), aiming to enhance model generalization for unseen modality combinations. To address this challenge, we propose the model named $\textbf{C}$ontrastive $\textbf{M}$ixed $\textbf{P}$rompt $\textbf{L}$earning ($\textsf{CMPL}$) for IMSAUMC. It introduces a label-guided contrastive feature learning mechanism to learn robust and discriminative cross-modal representations. Additionally, we design modality-combination prompts with a soft router to facilitate better learning of various modality combinations. Furthermore, we introduce three prompt contrastive learning strategies, which enable effective learning of prompts corresponding to unseen modality combinations, thereby significantly strengthening the model's generalization capabilities in diverse testing scenarios. Extensive experiments on three widely used datasets demonstrate that $\textsf{CMPL}$ achieves more than a 5% improvement in accuracy compared to state-of-the-art approaches.
