Search papers, labs, and topics across Lattice.
This paper introduces the Multimodal Intra- and Cross-Context Vision Transformer (MICViT), a novel 3D vision transformer designed to effectively integrate modality-specific representations and cross-modal interactions in brain MRI analysis. By employing four distinct attention mechanisms, MICViT captures both local and global contexts within and across modalities, addressing the challenges posed by high-dimensional 3D data and limited sample sizes in neuroimaging. Evaluated on brain age prediction across three diverse datasets, MICViT significantly outperforms existing CNN and transformer baselines, particularly benefiting from the inclusion of multiple MRI modalities.
Explicitly modeling intra- and cross-modal interactions in brain MRI can lead to substantial performance improvements, with MICViT outperforming traditional methods by leveraging multimodal inputs.
Brain MRI poses a fundamental challenge for machine learning: models must learn from high-dimensional 3D data spanning multiple co-registered modalities, despite the limited sample sizes typical of neuroimaging studies relative to the diversity in anatomy, pathology, and acquisition conditions. While multimodal imaging provides complementary information critical for clinical interpretation, effectively integrating these signals remains difficult. We propose Multimodal Intra- and Cross-Context Vision Transformer (MICViT), a 3D vision transformer that explicitly models both modality-specific representations and cross-modal interactions across local and global contexts. Concretely, MICViT combines four attention mechanisms: modality-specific local and global attention for intra-modal feature learning, and cross-modal local and global attention to capture interactions between modalities. We evaluate MICViT on brain age prediction across three heterogeneous datasets (UK Biobank, n=41,404; SOOP, n=1,062; Cam-CAN, n=613) using multiple MRI modalities (e.g. T1, FLAIR, DWI, SWI). MICViT consistently outperforms state-of-the-art CNN and transformer baselines in 3D settings. Notably, it benefits more strongly from multimodal inputs, yielding larger performance gains as additional modalities are incorporated. These results demonstrate that explicitly modeling intra- and cross-modal interactions is key to unlocking the full potential of multimodal brain MRI, highlighting a promising direction for representation learning in neuroimaging.
