Total: 1
Sparse training is often adopted in cross-device federated learning (FL) environments where constrained devices collaboratively train a machine learning model on private data by exchanging pseudo-gradients across heterogeneous networks. Although sparse training methods can reduce communication overhead and computational burden in FL, they are often not used in practice for the following key reasons: (1) data heterogeneity impacts more clients’ consensus on sparse, compared to dense, models, requiring training for longer; (2) a lack of sufficient plasticity to adapt to never-seen data distributions, crucial in cross-device FL; (3) requiring additional hyperparameters, which are notably challenging to tune in FL. This paper presents SparsyFed, a practical federated sparse training method that critically addresses all the aforementioned problems. Previous works have only managed to solve one, or perhaps two of these challenges, and at the expense of introducing new trade-offs, such as clients’ consensus on masks versus sparsity pattern plasticity. We show that SparsyFed simultaneously (1) can produce 95% sparse models, with negligible degradation in accuracy, while only needing a single hyperparameter, (2) achieves a per-round weight regrowth 200 times smaller than previous methods, and (3) still offers plasticity under this sparse design, by outperforming all the baselines at adapting to never-seen data distributions.