Search papers, labs, and topics across Lattice.
This paper introduces FM$^2$, a unified framework for training federated foundation models specifically tailored for heterogeneous multimodal medical imaging, addressing the challenge of Imaging Modality Heterogeneity. By employing dual Mixture-of-Experts modules and a Heterogeneous Modality Alignment regularizer, FM$^2$ achieves significant improvements in model fidelity and generalization across various medical imaging tasks while adhering to privacy regulations. Experimental results on the MIMH benchmark and real-world datasets demonstrate FM$^2$'s superiority over existing federated approaches, highlighting its effectiveness in cross-modality representation transfer and classification tasks.
Federated learning for medical imaging just got a major upgrade, enabling seamless representation transfer across disjoint modalities while preserving patient privacy.
Building foundation models for medical imaging requires pooling data across institutions, yet privacy regulations prohibit centralized aggregation. Existing Federated Foundation Models either fine-tune natural-image models with poor medical-domain transfer, or train from scratch within a single modality, lacking the flexibility to unify tasks. We identify an under-explored challenge, Imaging Modality Heterogeneity, where clients operate under two structural regimes: Overlapped (shared modalities with heterogeneous label distributions) and Non-overlapped (fully disjoint modalities per client). We propose FM$^2$, a unified framework that trains the core backbone from scratch to preserve medical domain fidelity while optionally incorporating biomedical pretrained encoders for vision-language alignment. FM$^2$ equips each client with dual Mixture-of-Experts modules (a Class-wise MoE for personalized category knowledge and a Domain-wise MoE for shared cross-modality representations), coupled with a Heterogeneous Modality Alignment (HMA) regularizer that explicitly aligns modality-specific expert parameters, admitting provable $O(1/\sqrt{T})$ convergence and generalization guarantees. FM$^2$ further incorporates Caption-Enhanced Learning (CEL), where locally retained GPT-4o-generated captions serve as a textual semantic bridge enabling representation transfer across clients with disjoint modalities, and demonstrates extensibility to Federated Medical VQA. Experiments on our MIMH benchmark (classification and CEL) and real-world medical VQA datasets confirm consistent superiority over state-of-the-art federated baselines and strong out-of-modality generalization across all three tasks.
