Search papers, labs, and topics across Lattice.
The paper introduces ML-ECS, a collaborative multimodal learning framework designed for edge-cloud synergies, addressing challenges of modality and model-structure heterogeneity in real-world edge environments. ML-ECS employs cross-modal contrastive learning, adaptive multimodal tuning, modality-aware model aggregation, and SLM-enhanced CCL to enable joint training between server-based and heterogeneous edge models while preserving privacy. Experiments demonstrate that ML-ECS outperforms state-of-the-art baselines across various multimodal tasks, achieving significant improvements in Rouge-LSum and demonstrating high communication efficiency by transmitting only low-rank LoRA parameters.
Achieve up to 12% improvement in Rouge-LSum for multimodal tasks in edge-cloud settings by jointly training heterogeneous edge models with a server model, even with varying modality availability.
Edge-cloud synergies provide a promising paradigm for privacy-preserving deployment of foundation models, where lightweight on-device models adapt to domain-specific data and cloud-hosted models coordinate knowledge sharing. However, in real-world edge environments, collaborative multimodal learning is challenged by modality heterogeneity (different modality combinations across domains) and model-structure heterogeneity (different modality-specific encoders/fusion modules. To address these issues, we propose ML-ECS, a collaborative multimodal learning framework that enables joint training between a server-based model and heterogeneous edge models. This framework consists of four components: (1) cross-modal contrastive learning (CCL) to align modality representations in a shared latent space, (2) adaptive multimodal tuning (AMT) to preserve domain-specific knowledge from local datasets, (3) modality-aware model aggregation (MMA) to robustly aggregate while mitigating noise caused by missing modalities, and (4) SLM-enhanced CCL (SE-CCL) to facilitate bidirectional knowledge transfer between cloud and edge. Experimental results on various multimodal tasks show that \pname consistently outperform state-of-the-art baselines under varying modality availability, achieving improvements of 5.44% to 12.08% in Rouge-LSum and improving both client- and server-side performance. In addition, by communicating only low-rank LoRA parameters and fused representations, ML-ECS achieves high communication efficiency, requiring only 0.65% of the total parameter volume.
