Search papers, labs, and topics across Lattice.
This paper introduces UniMod, a novel framework designed to enhance multi-modal medical diagnosis by addressing shortcut learning in models that combine medical images and clinical text. By enforcing independent predictions from each modality while also incorporating cross-modality and within-modality alignment, UniMod ensures that both text and image features are effectively utilized for diagnosis. The framework demonstrates significant performance improvements, achieving AUC scores of 0.850 on Harvard-Glaucoma and 0.966 on CheXpert Plus, surpassing existing methods by notable margins.
UniMod forces models to learn from both images and text independently, eliminating shortcut learning and achieving state-of-the-art diagnostic accuracy.
Multi-modal learning combining medical images and clinical text is promising for disease diagnosis. However, standard multi-modal training leads to shortcut learning: models exploit the easier modality (e.g., diagnostic cues in text) while neglecting harder-to-learn features (e.g., subtle visual patterns). We propose UniMod, a framework that mitigates shortcut learning by requiring each modality to predict the diagnosis on its own. It supervises image-only, text-only, and multi-modal classification simultaneously, so each modality must extract diagnostic features. We add cross-modality alignment for knowledge transfer and within-modality supervised contrastive alignment over same-diagnosis patients. On Harvard-Glaucoma, UniMod reaches 0.850 AUC, outperforming OGM-GE and Gradient Blending by 1.6-1.8%; on CheXpert Plus, it reaches 0.966 AUC, surpassing them by over 5%. UniMod also extends to 5-class multi-label diagnosis without architectural change, improving mean AUC by 0.097 over CGGM.
