Search papers, labs, and topics across Lattice.
This paper introduces UniBrain, a unified multimodal model designed for the analysis of brain MRI data, addressing the critical issue of missing modalities in clinical settings. By employing a joint imaging modality imputation strategy alongside medical reasoning capabilities, the model effectively enhances the understanding and diagnosis of brain diseases. Experimental results show that UniBrain significantly outperforms existing methods in terms of imputation accuracy and diagnostic performance, even in the presence of incomplete data.
UniBrain achieves remarkable performance in brain MRI analysis, even when faced with significant data incompleteness, revolutionizing how we approach medical imaging challenges.
Multimodal large language models (MLLMs) hold great potential for medicine, as they inherit knowledge from LLM and allow multiple data modalities to be integrated, analysed and interpreted in natural language. However, the field of medical MLLMs is constrained by non-trivial challenges, notably the scarcity of high-quality training data and the frequent occurrence of missing data in the real-world clinical setting. Here, we propose a novel unified multimodal model, UniBrain, for brain magnetic resonance image (MRI) analysis. To address potential missing brain MRI modalities, we employ a unified training strategy to perform joint imaging modality imputation and brain image understanding. During training, an interleaved and description-enriched data flow is constructed to train the model in an autoregressive manner, enabling medical reasoning with generated multimodal data. A self-alignment strategy is introduced to leverage dense image embeddings to learn fine-grained anatomical features without requiring detailed image captions. Furthermore, we propose a dynamic hidden state mechanism to alleviate the exposure bias during long-context multimodal inference. Extensive experiments on multi-disease brain MRI dataset demonstrate that UniBrain achieves high performance for brain image imputation, understanding, and disease diagnosis under various extents of modality incompleteness.
