Search papers, labs, and topics across Lattice.
This paper revisits the readout mechanism for multimodal sentiment analysis (MSA) by introducing a discriminative approach using the Thinker module of the Qwen2.5-Omni-7B LLM, which maps hidden states directly to continuous sentiment scores. The authors demonstrate that this method significantly improves accuracy and efficiency compared to traditional generative readout, achieving state-of-the-art results on the CMU-MOSI and CMU-MOSEI datasets without the need for task-specific feature engineering. Their findings reveal that the choice of readout mechanism is critical, as the discriminative approach reduces mean absolute error and latency while enhancing stability across multiple seeds.
Discriminative readout from an omni-modal LLM slashes MSA error rates and latency, outperforming traditional generative methods by a wide margin.
Multimodal sentiment analysis (MSA) infers human affect from language, acoustic, and visual signals. Recent methods increasingly adapt large multimodal models (LMMs) via generative readout: prompting the model to emit a sentiment score as a text string. While convenient, this ties continuous regression to discrete autoregressive decoding, incurring unmeasured costs. We revisit this readout mechanism and propose a discriminative formulation built on the Thinker module of a native omni-modal LLM (Qwen2.5-Omni-7B). Instead of text decoding, we map the final-layer hidden state of the last non-padding token to a continuous score via a lightweight regression head in a single forward pass. Using 4-bit quantization and low-rank adaptation (QLoRA), the entire 7B pipeline -- including video and audio processing -- trains on a single consumer GPU (RTX 5090, 32 GB) with 10-21 GB peak memory and 1.14% trainable parameters. Through a controlled comparison fixing the backbone, data, and LoRA configuration, we isolate the impact of the readout. On CMU-MOSI and CMU-MOSEI, our discriminative readout reaches state-of-the-art accuracy without task-specific feature engineering (MOSI: MAE 0.551, Corr 0.888; MOSEI: MAE 0.506, Corr 0.790) and exhibits strong multi-seed stability. In contrast, the generative readout -- even after equivalent supervised training -- more than doubles the mean absolute error, yields unparsable or out-of-range outputs (2.8% zero-shot), and suffers from higher latency. Modality ablations reveal a text-dominant regime on CMU-MOSI. Our findings indicate that how an LMM is read out is as consequential as how it is trained, demonstrating that a discriminative readout offers a more accurate, efficient, and reliable alternative for continuous MSA.
