Search papers, labs, and topics across Lattice.
This paper rigorously analyzes the robustness of distributed self-supervised learning (D-SSL) frameworks in the context of non-IID data, revealing that Masked Image Modeling (MIM) outperforms Contrastive Learning (CL) in heterogeneous environments. The study establishes that increased average network connectivity enhances the robustness of decentralized SSL, suggesting that federated learning (FL) is comparably robust to decentralized learning (DecL). Additionally, the introduction of MAR loss, which incorporates local-to-global alignment regularization, demonstrates practical benefits aligned with the theoretical findings through extensive experiments.
MIM's superior robustness against non-IID data could redefine the benchmarks for distributed self-supervised learning frameworks.
Recent research has introduced distributed self-supervised learning (D-SSL) approaches to leverage vast amounts of unlabeled decentralized data. However, D-SSL faces the critical challenge of data heterogeneity, and there is limited theoretical understanding of how different D-SSL frameworks respond to this challenge. To fill this gap, we present a rigorous theoretical analysis of the robustness of D-SSL frameworks under non-IID (non-independent and identically distributed) settings. Our results show that pre-training with Masked Image Modeling (MIM) is inherently more robust to heterogeneous data than Contrastive Learning (CL), and that the robustness of decentralized SSL increases with average network connectivity, implying that federated learning (FL) is no less robust than decentralized learning (DecL). These findings provide a solid theoretical foundation for guiding the design of future D-SSL algorithms. To further illustrate the practical implications of our theory, we introduce MAR loss, a refinement of the MIM objective with local-to-global alignment regularization. Extensive experiments across model architectures and distributed settings validate our theoretical insights, and additionally confirm the effectiveness of MAR loss as an application of our analysis.
