Search papers, labs, and topics across Lattice.
This paper introduces a Continuous Reliability Spectrum to model modality missingness and quality degradation in multimodal sentiment analysis (MSA). They propose QA-MoE, a Mixture-of-Experts architecture that uses self-supervised aleatoric uncertainty to quantify modality reliability and guide expert routing. Experiments show QA-MoE achieves state-of-the-art performance across various degradation scenarios, demonstrating a "One-Checkpoint-for-All" property.
Existing multimodal sentiment analysis models crumble under real-world noise, but QA-MoE leverages uncertainty to dynamically route inputs, achieving robust performance across a continuous spectrum of data quality.
Multimodal Sentiment Analysis (MSA) aims to infer human sentiment from textual, acoustic, and visual signals. In real-world scenarios, however, multimodal inputs are often compromised by dynamic noise or modality missingness. Existing methods typically treat these imperfections as discrete cases or assume fixed corruption ratios, which limits their adaptability to continuously varying reliability conditions. To address this, we first introduce a Continuous Reliability Spectrum to unify missingness and quality degradation into a single framework. Building on this, we propose QA-MoE, a Quality-Aware Mixture-of-Experts framework that quantifies modality reliability via self-supervised aleatoric uncertainty. This mechanism explicitly guides expert routing, enabling the model to suppress error propagation from unreliable signals while preserving task-relevant information. Extensive experiments indicate that QA-MoE achieves competitive or state-of-the-art performance across diverse degradation scenarios and exhibits a promising One-Checkpoint-for-All property in practice.
Tsinghua AI