Search papers, labs, and topics across Lattice.
This paper introduces Multimodal Semantic-Aware Contrastive Learning (MseaCL), a novel framework designed to address the issue of false negatives in 3D medical imaging by leveraging semantic similarities between radiology reports. By incorporating semantic attributes into the contrastive learning process, MseaCL significantly enhances the alignment of multimodal representations, particularly in the context of pediatric brain MRI scans. The framework demonstrates a substantial improvement in downstream tasks, achieving at least a 22.6% increase in AUC for pediatric brain tumor molecular classification, underscoring its effectiveness in clinical applications.
False negatives in medical imaging can be mitigated by a new contrastive learning approach that leverages semantic similarities, leading to a 22.6% boost in diagnostic accuracy.
Multimodal Contrastive Learning (CL) has shown significant performance in aligning representations across various data modalities and improving downstream tasks, especially in healthcare. It works by minimizing the distance between matched (positive) data modalities, while maximizing the distance between mismatched (negative) samples. Traditional CL frameworks typically assume instance-based correspondence within data batches, treating all non-paired samples as negatives. However, this assumption often fails in medical settings, where samples may share high-level semantic attributes, leading to false negatives that degrade representation quality. In this paper, we propose Multimodal Semantic-Aware Contrastive Learning (MseaCL), a CL framework trained on a pediatric cohort of 3D brain magnetic resonance imaging (MRI) scans and radiology reports. The goal of this framework is to mitigate the impact of semantically similar false negative samples by incorporating semantic similarity between radiology reports, as a guiding signal during the learning process. Our results indicate that applying this framework as a pretraining stage can achieve notable improvements in downstream tasks, e.g., at least a 22.6\% increase in the area under the receiver operating characteristic curve (AUC) of pediatric brain tumor molecular classification, demonstrating its potential for more robust and semantically aligned multimodal representations in clinical applications.
