Search papers, labs, and topics across Lattice.
The paper introduces Semi-Push-Pull Supervised Contrastive Learning (SPP-SCL) to address inconsistent intra-modal and inter-modal sentiment relationships in Image-Text Sentiment Analysis (ITSA). SPP-SCL employs a two-step strategy: intra-modal supervised contrastive learning to pull related intra-modal features, followed by conditional inter-modal supervised contrastive learning to push away unrelated inter-modal features. Experiments on three ITSA and sarcasm detection datasets demonstrate that SPP-SCL outperforms existing methods, indicating improved sentiment discrimination.
Achieve state-of-the-art results in image-text sentiment analysis by balancing intra-modal and inter-modal relationships with a novel "push-pull" contrastive learning strategy.
Existing Image-Text Sentiment Analysis (ITSA) methods may suffer from inconsistent intra-modal and inter-modal sentiment relationships. Therefore, we develop a method that balances before fusing to solve the issue of vision-language imbalance intra-modal and inter-modal sentiment relationships; that is, a Semi-Push-Pull Supervised Contrastive Learning (SPP-SCL) method is proposed. Specifically, the method is implemented using a novel two-step strategy, namely first using the proposed intra-modal supervised contrastive learning to pull the relationships between the intra-modal and then performing a well-designed conditional execution statement. If the statement result is false, our method will perform the second step, which is inter-modal supervised contrastive learning to push away the relationships between inter-modal. The two-step strategy will balance the intra-modal and inter-modal relationships to achieve the purpose of relationship consistency and finally perform cross-modal feature fusion for sentiment analysis and detection. Experimental studies on three public image-text sentiment and sarcasm detection datasets demonstrate that SPP-SCL significantly outperforms state-of-the-art methods by a large margin and is more discriminative in sentiment.
