Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the role of reliability scores in multimodal systems by introducing a diagnostic method to assess their influence on model decisions during inference. Through experiments on stress recognition and sentiment analysis datasets, the authors demonstrate that permuting reliability scores does not affect model performance, suggesting that these scores do not actively guide decision-making. However, when reliability signals accurately identify the correct modality, significant performance improvements are observed, highlighting the conditional dependence of multimodal fusion on reliable predictions.
Reliability scores in multimodal systems may not influence decisions as expected, revealing a disconnect between potential and actual performance gains. WHY_IT MATTERS: This insight challenges assumptions about the utility of reliability metrics in multimodal systems, potentially reshaping how researchers design and evaluate these models.
Many multimodal systems estimate the reliability of each modality and weight their contributions to the final prediction. However, it remains unclear whether these scores influence model decisions or merely correlate with performance. We propose a simple diagnostic to test whether reliability information is used during inference. After training, the model and inputs are fixed while reliability scores are permuted across test examples. If predictions depend on these scores, performance should degrade. Experiments on StressID for stress recognition and CMU-MOSEI for sentiment analysis show that permuting reliability scores leaves performance unchanged despite substantial potential gains from selecting the best modality per example. In positive controls where reliability signals identify the correct modality, the same frozen fusion rules yield significant improvements, indicating that reliability signals influence fused decisions only when they reliably predict unimodal correctness.