Complement Sparsification: Low-Overhead Model Pruning for Federated Learning

by   Xiaopeng Jiang, et al.

Federated Learning (FL) is a privacy-preserving distributed deep learning paradigm that involves substantial communication and computation effort, which is a problem for resource-constrained mobile and IoT devices. Model pruning/sparsification develops sparse models that could solve this problem, but existing sparsification solutions cannot satisfy at the same time the requirements for low bidirectional communication overhead between the server and the clients, low computation overhead at the clients, and good model accuracy, under the FL assumption that the server does not have access to raw data to fine-tune the pruned models. We propose Complement Sparsification (CS), a pruning mechanism that satisfies all these requirements through a complementary and collaborative pruning done at the server and the clients. At each round, CS creates a global sparse model that contains the weights that capture the general data distribution of all clients, while the clients create local sparse models with the weights pruned from the global model to capture the local trends. For improved model performance, these two types of complementary sparse models are aggregated into a dense model in each round, which is subsequently pruned in an iterative process. CS requires little computation overhead on the top of vanilla FL for both the server and the clients. We demonstrate that CS is an approximation of vanilla FL and, thus, its models perform well. We evaluate CS experimentally with two popular FL benchmark datasets. CS achieves substantial reduction in bidirectional communication, while achieving performance comparable with vanilla FL. In addition, CS outperforms baseline pruning mechanisms for FL.


FedMR: Fedreated Learning via Model Recombination

As a promising privacy-preserving machine learning method, Federated Lea...

Efficient Federated Learning with Enhanced Privacy via Lottery Ticket Pruning in Edge Computing

Federated learning (FL) is a collaborative learning paradigm for decentr...

Model Pruning Enables Localized and Efficient Federated Learning for Yield Forecasting and Data Sharing

Federated Learning (FL) presents a decentralized approach to model train...

FedNet2Net: Saving Communication and Computations in Federated Learning with Model Growing

Federated learning (FL) is a recently developed area of machine learning...

Truthful Incentive Mechanism for Federated Learning with Crowdsourced Data Labeling

Federated learning (FL) has emerged as a promising paradigm that trains ...

Multi-Armed Bandit Based Client Scheduling for Federated Learning

By exploiting the computing power and local data of distributed clients,...

Does Federated Learning Really Need Backpropagation?

Federated learning (FL) is a general principle for decentralized clients...

Please sign up or login with your details

Forgot password? Click here to reset