Search papers, labs, and topics across Lattice.
This paper addresses the issue of strong-modality collapse in multimodal learning, where the performance of the dominant modality degrades during integration with weaker modalities. By introducing Inverted Asymmetric Fusion (IAF), the authors preserve the integrity of the dominant modality while allowing weaker modalities to enhance their contextual relevance through Modality-Aware Knowledge Distillation. The results demonstrate that IAF maintains the dominant modality's accuracy and improves overall performance on multimodal benchmarks, outperforming the best unimodal baseline by up to 8.25%.
Strong-modality collapse can degrade dominant modalities by over 18%, but Inverted Asymmetric Fusion preserves performance while enhancing weaker modalities.
Fusing multiple modalities is expected to improve model performance. However, on the MultiHuSE dataset, early, late, and symmetric attention fusion often fail to outperform the best unimodal baseline (text). Pathway isolation of a symmetric attention fusion model reveals that the text-pathway accuracy drops from 74.9% to 56.4% after fusion in one such setting, indicating that the dominant modality can be degraded during integration. We term this strong-modality collapse and argue that it helps explain why some multimodal models fail to surpass unimodal baselines. We propose Inverted Asymmetric Fusion (IAF), which avoids forcing mutual attention across modalities. The dominant modality is preserved by passing through fusion unchanged, while weaker modalities attend to it as a contextual anchor. Before fusion, weaker modalities are strengthened using Modality-Aware Knowledge Distillation. We evaluate IAF on three benchmarks with different modality hierarchies: text-dominant datasets (MultiHuSE, UR-FUNNY) and an audio-visual-dominant dataset (MUStARD). Pathway isolation shows that IAF preserves the dominant modality's internal accuracy at its unimodal ceiling across all tested configurations, whereas symmetric fusion degrades it by up to 18.5% on MultiHuSE. IAF improves over the strongest unimodal baseline by up to 8.25%.
