Search papers, labs, and topics across Lattice.
This paper investigates the impact of update masking on regularization in federated learning, particularly in non-IID settings. The authors reveal that using client-specific masks during parameter updates weakens the optimizer's charge against sharp directions, with the degree of weakening inversely related to the cohort's gradient diversity. Experiments on CIFAR-10 demonstrate that while certain configurations can enhance regularization, they also lead to poor training performance, highlighting a trade-off in the approach.
Regularization in federated learning can be significantly influenced by the choice of update masks, revealing a critical trade-off between generalization and training efficacy.
Federated learning on non-IID data seeks flat minima to generalize across clients, and existing methods borrow sharpness-aware minimization from centralized training. There is a second way to reach flat minima, in which the regularization comes for free from noise added to the parameter updates, and it has never been carried over to the federated setting. We show the reason. Masking charges the optimizer for moving in sharp directions. We prove that when each client draws its own mask, federated averaging weakens that charge by exactly the cohort size, and that giving every client the same mask brings it back by a factor equal to the inverse gradient diversity of the cohort. In our experiment setting on CIFAR-10, that factor is 1.19 out of a possible 10. Turning off minibatch sampling raises it to 8.96, while changing data heterogeneity a hundredfold leaves it between 1.17 and 1.50. The configurations keeping the regularization train far too poorly to use.
