Search papers, labs, and topics across Lattice.
The paper introduces FedAFD, a multimodal federated learning framework designed to address challenges in personalized client performance, modality/task discrepancies, and model heterogeneity. FedAFD employs a bi-level adversarial alignment strategy on the client side to align local and global representations within and across modalities, along with a granularity-aware fusion module for adaptive knowledge integration. On the server side, a similarity-guided ensemble distillation mechanism aggregates client representations based on feature similarity and distills the fused knowledge into the global model, improving performance and efficiency in both IID and non-IID settings.
By aligning local and global representations with adversarial training and distilling knowledge based on feature similarity, FedAFD achieves superior performance in multimodal federated learning scenarios where data and models are heterogeneous.
Multimodal Federated Learning (MFL) enables clients with heterogeneous data modalities to collaboratively train models without sharing raw data, offering a privacy-preserving framework that leverages complementary cross-modal information. However, existing methods often overlook personalized client performance and struggle with modality/task discrepancies, as well as model heterogeneity. To address these challenges, we propose FedAFD, a unified MFL framework that enhances client and server learning. On the client side, we introduce a bi-level adversarial alignment strategy to align local and global representations within and across modalities, mitigating modality and task gaps. We further design a granularity-aware fusion module to integrate global knowledge into the personalized features adaptively. On the server side, to handle model heterogeneity, we propose a similarity-guided ensemble distillation mechanism that aggregates client representations on shared public data based on feature similarity and distills the fused knowledge into the global model. Extensive experiments conducted under both IID and non-IID settings demonstrate that FedAFD achieves superior performance and efficiency for both the client and the server.
