Search papers, labs, and topics across Lattice.
This paper introduces MGSI, a multi-granularity sentiment integration framework designed to enhance LLM-based multimodal sentiment analysis by effectively incorporating audio and visual signals across varying temporal scales. By encoding these signals at short-, medium-, and long-range intervals, MGSI preserves both local variations and global trends, which are crucial for accurate sentiment prediction. The framework significantly outperforms existing frozen-LLM baselines and competes well with advanced multimodal methods, demonstrating the importance of fine-grained affective information in sentiment analysis.
MGSI reveals that integrating audio and visual sentiment cues at multiple temporal scales can dramatically enhance sentiment analysis accuracy in LLMs.
Multimodal sentiment analysis (MSA) aims to predict sentiment polarity and intensity from heterogeneous inputs such as text, audio, and vision. While large language models (LLMs) offer strong semantic priors for MSA, effectively incorporating audio and visual signals effectively remains challenging. A key challenge is that audio and visual sentiment cues evolve over different temporal scales, yet many LLM-based methods compress these signals through shallow projection or coarse pooling before fusing them with text, which can weaken cross-modal alignment and erase fine-grained affective information. We propose MGSI, a multi-granularity sentiment integration framework for LLM-based MSA. MGSI first encodes audio and visual streams at short-, medium-, and long-range temporal scales, preserving both local variations and global affective trends. It then refines non-text features through text-guided alignment, and applies polarity- and intensity-aware enhancement to better handle ambiguous and near-neutral samples. The resulting multimodal representation is finally compressed into a small set of pseudo-tokens for efficient conditioning of a frozen LLM. Experiments on four public benchmarks show that MGSI substantially outperforms frozen-LLM baselines and remains competitive with strong multimodal methods. Further ablation and sensitivity analyses support the effectiveness of multi-granularity temporal modeling, text-guided refinement, and adaptive sentiment calibration.
