Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper explores the use of entropy-regularized probabilistic gates in federated learning to enhance sparse model discovery in scenarios with limited data. By integrating an L0 constraint and promoting uncertainty in model parameters, the proposed method effectively mitigates the challenges posed by data heterogeneity and partial client participation. Experimental results demonstrate that this approach outperforms traditional methods like federated iterative hard thresholding and pruning after dense federated averaging, both in terms of statistical performance and sparsity recovery accuracy.
Entropy regularization in federated learning can significantly enhance model performance and sparsity recovery, even in data-scarce environments.
Federated Learning (FL) is a distributed machine learning (ML) paradigm with collaboration among multiple clients without sharing data. FL is challenging under data heterogeneity and partial client participation. Learning sparse models is useful for communication and computational efficiency in FL, but it is especially difficult in the small-sample high-dimensional regime (d >> N) where optimization can yield parameter configurations that fail to generalize to unseen test data. While magnitude-based pruning doesn't account for uncertainty exploration in the parameter space, a formulation with probabilistic gates and an L0 constraint allows sampling from competing sparse configurations during training. In this work, we study entropy regularization of gate distributions as a mechanism to maintain uncertainty in sparse federated optimization by preventing early commitment to sparse support. We examine its impact under data heterogeneity, client participation heterogeneity, and sparsity. Experiments on synthetic and real-world benchmarks show consistent improvements over federated iterative hard thresholding (Fed-IHT) and pruning after dense federated averaging (FedAvg) training, both in statistical performance on test data and in sparsity recovery accuracy.