Search papers, labs, and topics across Lattice.
This study explores the efficacy of deep ensembles of unimodal networks for multimodal classification, demonstrating that they can outperform traditional late-fusion methods even in scenarios with significant modality imbalance. By systematically comparing these ensembles to state-of-the-art late-fusion and intermediate-fusion techniques, the authors reveal that ensembles consistently deliver superior performance without the need for explicit modality fusion. Additionally, they introduce a heuristic for selecting the optimal number of models per modality in an ensemble, which is validated through both synthetic and real-world datasets, leading to insights on scaling performance with ensemble size.
Unimodal deep ensembles can outperform late-fusion networks in multimodal classification, even under severe modality imbalance.
In multimodal classification, late-fusion approaches classify concatenated modality-specific features extracted by unimodal neural networks. When modality imbalance is pronounced, various regularization techniques have been proposed to balance the learning process and overcome the inferior performance of late-fusion networks. In contrast, this work demonstrates that multimodal data can be effectively classified without any explicit modality fusion, using deep ensembles of unimodal networks. We systematically compare deep ensembles to late-fusion networks at equal parameter count and show that ensembles consistently outperform state-of-the-art late-fusion methods designed to address modality imbalance. This advantage also holds over intermediate-fusion techniques we evaluated and over hybrid methods that combine unimodal and multimodal predictions. We propose and empirically validate a method for selecting the number of models per modality in an ensemble, avoiding computationally expensive exhaustive search. Under extreme modality imbalance and small ensemble sizes, the heuristic indicates that ensembles of unimodal models trained solely on the stronger modality are preferable; as the ensemble scales up, incorporating models from the weaker modality becomes beneficial. Both predictions align with our empirical findings. To systematically explore the challenges of optimizing multimodal models, we propose a synthetic multimodal framework that allows control over both the number of modalities and their predictive strength; our findings are consistent across synthetic and real-world datasets. Finally, by fitting scaling laws to bimodal datasets, we estimate the asymptotic performance of ensembles.
