Search papers, labs, and topics across Lattice.
MissBench is introduced as a benchmark and framework to evaluate multimodal affective models under imbalanced missing modality conditions, addressing the limitations of standard evaluations that assume equal availability of textual, acoustic, and visual modalities. The benchmark includes shared and imbalanced missing-rate protocols on four sentiment and emotion datasets, along with two diagnostic metrics: Modality Equity Index (MEI) and Modality Learning Index (MLI). Experiments on representative method families reveal that models robust under shared missing rates can exhibit modality inequity and optimization imbalance under imbalanced conditions, highlighting the utility of MissBench for stress-testing and analyzing multimodal models.
Multimodal models that seem robust can still fail when some modalities are systematically missing, a problem MissBench exposes with new metrics for modality equity and learning balance.
Multimodal affective computing underpins key tasks such as sentiment analysis and emotion recognition. Standard evaluations, however, often assume that textual, acoustic, and visual modalities are equally available. In real applications, some modalities are systematically more fragile or expensive, creating imbalanced missing rates and training biases that task-level metrics alone do not reveal. We introduce MissBench, a benchmark and framework for multimodal affective tasks that standardizes both shared and imbalanced missing-rate protocols on four widely used sentiment and emotion datasets. MissBench also defines two diagnostic metrics. The Modality Equity Index (MEI) measures how fairly different modalities contribute across missing-modality configurations. The Modality Learning Index (MLI) quantifies optimization imbalance by comparing modality-specific gradient norms during training, aggregated across modality-related modules. Experiments on representative method families show that models that appear robust under shared missing rates can still exhibit marked modality inequity and optimization imbalance under imbalanced conditions. These findings position MissBench, together with MEI and MLI, as practical tools for stress-testing and analyzing multimodal affective models in realistic incomplete-modality settings.For reproducibility, we release our code at: https://anonymous.4open.science/r/MissBench-4098/
