Search papers, labs, and topics across Lattice.
This paper tackles the challenge of training multimodal sentiment analysis models where the text modality often dominates, suppressing acoustic and visual modalities. They introduce a Conflict-aware Penalty (CP) to mitigate gradient norm conflicts between modalities and a Statistical Loss (SL) to align predicted and empirical distribution statistics. By preventing dominant modality gradients from interfering with the SL objective, the proposed framework achieves state-of-the-art performance on CMU-MOSI.
Text modality dominance is holding back multimodal models, but penalizing gradient conflicts and aligning predicted statistics can unlock synergistic training and state-of-the-art results.
Multimodal Sentiment Analysis (MSA) fuses text, acoustic, and visual streams to infer sentiment. Because pre-trained text encoders are far more expressive than their acoustic and visual counterparts, the text modality tends to dominate optimization, suppressing weaker modalities and inducing gradient norm conflicts that destabilize training. To address this, we propose a Conflict-aware Penalty (CP) that detects and penalizes gradient norm conflicts at each training step, and a Statistical Loss (SL) that aligns predicted distribution statistics with empirical input statistics. Crucially, CP prevents dominant modality gradients from interfering with the SL objective, enabling synergistic training within a unified framework incorporating adaptive modality encoding, gated cross-modal fusion, and unimodal auxiliary heads. Experiments on CMU-MOSI demonstrate state-of-the-art performance, with ablation studies confirming the effectiveness of each component.
