Search papers, labs, and topics across Lattice.
The paper investigates the modality gap phenomenon in multimodal medical representation learning, where CLIP-based contrastive losses lead to sparse and fragmented latent spaces, hindering true semantic alignment between radiology images and clinical text. They demonstrate that the modality gap exists in the medical domain and propose a modality-agnostic framework to address it. The proposed framework improves cross-modal retrieval and image captioning by ensuring better alignment of semantically related representations, regardless of modality.
Medical AI gets a boost: a new framework closes the "modality gap" in CLIP-based models, leading to better alignment between radiology images and clinical text.
In multimodal learning, CLIP has emerged as the de-facto approach for mapping different modalities into a shared latent space by bringing semantically similar representations closer while pushing apart dissimilar ones. However, CLIP-based contrastive losses exhibit unintended behaviors that negatively impact true semantic alignment, leading to sparse and fragmented latent spaces. This phenomenon, known as the modality gap, has been partially mitigated for standard text and image pairs but remains unknown and unresolved in more complex multimodal settings, such as the medical domain. In this work, we study this phenomenon in the latter case, revealing that the modality gap is present also in medical alignment, and we propose a modality-agnostic framework that closes this gap, ensuring that semantically related representations are more aligned, regardless of their source modality. Our method enhances alignment between radiology images and clinical text, improving cross-modal retrieval and image captioning.
