Search papers, labs, and topics across Lattice.
This paper introduces Flux, a novel multimodal federated learning framework that addresses the challenges of dual-axis modality missingness in health-sensing applications. By integrating modality-aware confidence tempering and gradient-decoupled private adaptation, Flux tailors predictions based on the quality and completeness of available modalities while maintaining robust shared model training. The framework outperforms existing methods across four multimodal datasets, achieving an average macro-F1 score improvement of 1.6 points over the strongest baseline.
Flux achieves unprecedented performance in multimodal federated learning by adapting predictions based on the specific confidence of available modalities, leading to significant improvements in accuracy.
Multimodal federated learning (FL) supports collaborative modeling in privacy-sensitive health-sensing and medical settings, but realistic deployments often exhibit dual-axis modality missingness: clients have different modality sets, and individual samples may contain only subsets of the modalities available locally. Existing methods typically address these two axes separately. We propose Flux, a multimodal federated learning framework built around two complementary components. First, modality-aware confidence tempering learns sample-specific confidence for each modality through mask-aware unimodal supervision and fuses the confidence estimates from observed modalities into a sample-adaptive temperature that adjusts predictive sharpness according to evidence quality and completeness. Second, gradient-decoupled private adaptation applies this temperature only to a client-private prediction pathway, while training the shared federated model with a standard, untempered objective. This enables sample-specific, client-local confidence adaptation without allowing confidence-dependent gradients to perturb shared representation learning. Across four multimodal datasets, Flux achieves the highest average macro-F1 on every dataset, outperforming the strongest dataset-specific baseline by 0.8~2.2 points and by 1.6 points on average. Additional analyses demonstrate favorable calibration, temperature sensitivity to both modality missingness and input corruption, and more stable shared optimization under private-only tempering. Our code is available at https://github.com/AdibaOrz/Flux.
