Search papers, labs, and topics across Lattice.
This paper challenges the conventional wisdom of balanced learning in multimodal models, arguing that prioritizing the performance-dominant modality leads to better multimodal performance. They identify that under-optimization stems from insufficient learning of this dominant modality. To address this, they introduce Performance-Dominant Modality Prioritization (PDMP), which uses unimodal performance ranking to modulate gradients, allowing the dominant modality to guide optimization. Experiments across diverse datasets demonstrate PDMP's effectiveness.
Forget balanced multimodal learning – letting the best modality lead the way actually unlocks better performance.
Multimodal learning has attracted increasing attention due to its practicality. However, it often suffers from insufficient optimization, where the multimodal model underperforms even compared to its unimodal counterparts. Existing methods attribute this problem to the imbalanced learning between modalities and solve it by gradient modulation. This paper argues that balanced learning is not the optimal setting for multimodal learning. On the contrary, imbalanced learning driven by the performance-dominant modality that has superior unimodal performance can contribute to better multimodal performance. And the under-optimization problem is caused by insufficient learning of the performance-dominant modality. To this end, we propose the Performance-Dominant Modality Prioritization (PDMP) strategy to assist multimodal learning. Specifically, PDMP firstly mines the performance-dominant modality via the performance ranking of the independently trained unimodal model. Then PDMP introduces asymmetric coefficients to modulate the gradients of each modality, enabling the performance-dominant modality to dominate the optimization. Since PDMP only relies on the unimodal performance ranking, it is independent of the structures and fusion methods of the multimodal model and has great potential for practical scenarios. Finally, extensive experiments on various datasets validate the superiority of PDMP.
