Search papers, labs, and topics across Lattice.
MIT CSAILCorresponding Authors.Department of Pathology and Laboratory MedicineHarvardNortheasternNorthwesternThese authors contribute equally to thisThese authors contributed equally toUPennYaleJun 16, 2026arXiv:2606.18123This study introduces MixTIME, a multimodal foundation model that utilizes a mixture-of-experts (MoE) architecture to integrate various image modalities for predicting immune biomarkers associated with the tumor immune microenvironment (TIME). By dynamically weighting expert contributions through a learnable router and employing a novel loss function, MixTIME achieves state-of-the-art performance in predicting multiplex immunofluorescence protein expression across multiple datasets. The model not only enhances the accuracy of biomarker predictions but also facilitates critical downstream applications such as survival prediction and tracking protein expression dynamics over time.
MixTIME reveals that integrating diverse image modalities can significantly enhance the precision of immune biomarker predictions in oncology, outperforming traditional single-modality approaches.
Predicting immune biomarkers associated with the tumor immune microenvironment (TIME) is critical for advancing precision oncology, yet existing approaches are largely limited to single image modalities and suffer from insufficient resolution and incomplete utilization of complementary clinical and biological information. Here we introduce MixTIME, a multimodal foundation model that leverages a mixture-of-experts (MoE) architecture to integrate pathology foundation models trained across distinct modalities: image only (UNIv2), image text (CONCHv1.5), and image transcriptomic (STPath) representations for pixel-level and slide-level prediction of multiplex immunofluorescence (mIF) protein expression from hematoxylin and eosin (HE) whole-slide images. MixTIME employs a learnable router to dynamically weight expert contributions and is trained with a distribution- and tendency-aware loss function. Benchmarked on two datasets of different scales, MixTIME achieves state-of-the-art performance across 17 protein markers as measured by correlation metrics. The predicted mIF profiles substantially enhance downstream tasks, including spatial domain identification, survival prediction, and AI-assisted pathology report generation validated by expert pathologists from multiple institutes across the world. Furthermore, MixTIME enables longitudinal tracking of protein expression dynamics across clinical time points and reveals protein gene interaction patterns linked to drug resistance and immune suppression in tumor microenvironments. Collectively, MixTIME provides a scalable framework for multimodal biomarker discovery and clinical translation in computational pathology.
