Search papers, labs, and topics across Lattice.
This paper demonstrates that decentralized learning can achieve centralized performance without sharing local datasets by using Empirical Risk Minimization with Relative-Entropy Regularization (ERM-RER) and a forward-backward communication scheme. The key is sharing locally obtained Gibbs measures, where each client uses the previous client's Gibbs measure as a reference, effectively encoding prior information. The analysis reveals that specific scaling of regularization factors with local sample sizes is crucial for matching centralized performance.
Decentralized learning can match centralized performance by sharing only Gibbs measures, not datasets, opening new avenues for privacy-preserving collaboration.
In this paper, it is shown, for the first time, that centralized performance is achievable in decentralized learning without sharing the local datasets. Specifically, when clients adopt an empirical risk minimization with relative-entropy regularization (ERM-RER) learning framework and a forward-backward communication between clients is established, it suffices to share the locally obtained Gibbs measures to achieve the same performance as that of a centralized ERM-RER with access to all the datasets. The core idea is that the Gibbs measure produced by client~$k$ is used, as reference measure, by client~$k+1$. This effectively establishes a principled way to encode prior information through a reference measure. In particular, achieving centralized performance in the decentralized setting requires a specific scaling of the regularization factors with the local sample sizes. Overall, this result opens the door to novel decentralized learning paradigms that shift the collaboration strategy from sharing data to sharing the local inductive bias via the reference measures over the set of models.
