Search papers, labs, and topics across Lattice.
This paper introduces Federated Continual Multimodal Learning (FedCMM), a framework designed to mitigate catastrophic forgetting in federated fine-tuning of Multimodal Large Language Models (MLLMs) across distributed networks. By integrating modality-aware elastic weight consolidation, lightweight local generative replay modules, and task-similarity-aware gradient aggregation, the approach enhances knowledge retention while adapting to evolving data streams. Experimental results show that FedCMM significantly improves accuracy and backward transfer compared to existing methods, highlighting its effectiveness for safety-sensitive applications like content moderation.
FedCMM achieves robust knowledge retention in MLLMs, outperforming traditional methods by effectively balancing adaptation and stability in dynamic environments.
Federated fine-tuning of Multimodal Large Language Models (MLLMs) across distributed networks enables privacy-sensitive adaptation to evolving data streams, yet a fundamental obstacle prevents robust deployment in dynamic environments: catastrophic forgetting, wherein sequential task updates erase previously acquired knowledge across visual, linguistic, and cross-modal representations. Addressing this challenge is especially critical for autonomous networked AI operating in safety-sensitive domains, such as content moderation, where reliable retention of prior knowledge underpins system integrity. To overcome this, we propose Federated Continual Multimodal Learning (FedCMM), a framework that embeds continual-learning safeguards into the federated optimization loop at three complementary levels. At the parameter level, modality-aware elastic weight consolidation computes separate Fisher information matrices for the vision encoder, language backbone, and cross-modal projector, providing granular, asymmetry-aware protection against modality-specific forgetting. At the data level, each client trains a lightweight local generative replay module to synthesize raw-data-free embedding-level multimodal replay tuples without any raw data sharing. At the aggregation level, Task-similarity-aware gradient aggregation autonomously filters and reweights client updates by gradient cosine similarity, suppressing conflicting directions and stabilizing the global learning trajectory. Extensive experiments on two benchmarks demonstrate that FedCMM consistently outperforms recent baselines on accuracy and backward transfer, confirming that holistic, modality-aware optimization enables robust evolutive adaptation across heterogeneous networked AI deployments.
