Search papers, labs, and topics across Lattice.
This paper investigates multimodal fusion of OCT and fundus photography for predicting Mean Deviation (MD) in ophthalmology, finding that naive fusion performs worse than unimodal models due to data imbalance and modality learning conflicts. To address this, they propose Re-M3Dr, a framework that enhances unimodal representation with supervised contrastive learning and stabilizes joint optimization using sharpness-aware gradient modulation. Experiments on public and private datasets demonstrate a 29% reduction in MSE compared to state-of-the-art multimodal methods.
Multimodal fusion can actually *hurt* performance in medical imaging tasks like predicting visual field loss, unless you carefully address data imbalance and modality conflict.
Mean Deviation (MD) is a critical metric for assessing visual field loss in ophthalmology. While previous work has focused solely on predicting MD from Optical Coherence Tomography (OCT), it is intuitive to assume that combining OCT with another imaging of fundus photography (FP) could improve performance, as two ophthalmic medical imaging provide complementary information. This is particularly expected when sophisticated multi-objective optimization is applied, as documented in common multimodal classification. Surprisingly, our investigations reveal that multimodal fusion in this medical imaging scenario performs worse than unimodal model. Through detailed analysis, we identify the root cause as a coupled imbalance between data distribution and modality learning conflict. This imbalance distorts the optimization landscape, leading to unstable training. To address this challenge, we propose the method of Rebalanced MultiModal Mean Deviation Regression (Re-M3Dr), a novel multimodal regression framework. We enhance unimodal representation through adaptive margin based supervised contrastive learning. Then, our framework stabilizes the joint optimization with the sharpness-aware gradient modulation. Experimental results on both public and private clinical datasets show average 29\% reduction in MSE compared to SOTA multimodal learning methods, demonstrating the superiority of Re-M3Dr. The code is available in the supplementary materials.
