Search papers, labs, and topics across Lattice.
This paper addresses the challenge of multimodal sentiment analysis by identifying representation misalignment between independently pretrained modality encoders as a key limitation. The authors propose a unified framework that utilizes vision-language models to convert visual content into structured textual descriptions, aligning heterogeneous modalities in a shared linguistic space. Their approach, which incorporates a hybrid learning strategy, consistently outperforms both unimodal and multimodal baselines, achieving state-of-the-art results and underscoring the importance of representation alignment in effective multimodal learning.
Representation misalignment is the hidden bottleneck in multimodal sentiment analysis, and aligning modalities before fusion can yield superior performance.
Multimodal affective analysis aims to understand human sentiment and emotion by jointly modeling heterogeneous modalities such as text and images. However, multimodal models often fail to consistently outperform strong text-only baselines, with performance varying significantly across fusion strategies. In this work, we identify representation misalignment between independently pretrained modality encoders as a key bottleneck for effective multimodal learning, and show through controlled experiments that alignment prior to fusion is often more important than fusion complexity. To address this issue, we propose a unified multimodal affective analysis framework that leverages vision-language models (VLMs) to convert visual content into structured textual descriptions, projecting heterogeneous modalities into a shared linguistic space and enabling interpretable text-centric reasoning. To further improve robustness, we introduce a hybrid learning strategy that combines semantic token selection with a batch-level uniformity regularization objective, encouraging a more dispersed and stable global feature space while mitigating noise introduced by VLM-generated descriptions. Experiments on multiple multimodal sentiment and emotion benchmarks show that our method consistently outperforms strong unimodal and multimodal baselines, achieving state-of-the-art performance. Our analysis further highlights the critical role of representation alignment in multimodal affective learning.
