DeepAI AI Chat
Log In Sign Up

Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?

by   Sheikh Shams Azam, et al.

In this paper, we question the rationale behind propagating large numbers of parameters through a distributed system during federated learning. We start by examining the rank characteristics of the subspace spanned by gradients across epochs (i.e., the gradient-space) in centralized model training, and observe that this gradient-space often consists of a few leading principal components accounting for an overwhelming majority (95-99 Motivated by this, we propose the "Look-back Gradient Multiplier" (LBGM) algorithm, which exploits this low-rank property to enable gradient recycling between model update rounds of federated learning, reducing transmissions of large parameters to single scalars for aggregation. We analytically characterize the convergence behavior of LBGM, revealing the nature of the trade-off between communication savings and model performance. Our subsequent experimental results demonstrate the improvement LBGM obtains in communication overhead compared to conventional federated learning on several datasets and deep learning models. Additionally, we show that LBGM is a general plug-and-play algorithm that can be used standalone or stacked on top of existing sparsification techniques for distributed model training.


page 30

page 33

page 34

page 36

page 37

page 39

page 40

page 42


Adaptive Parameterization of Deep Learning Models for Federated Learning

Federated Learning offers a way to train deep neural networks in a distr...

Communication-Efficient Federated Learning with Dual-Side Low-Rank Compression

Federated learning (FL) is a promising and powerful approach for trainin...

Fast-Convergent Federated Learning

Federated learning has emerged recently as a promising solution for dist...

Distilled One-Shot Federated Learning

Current federated learning algorithms take tens of communication rounds ...

Parallel Distributed Logistic Regression for Vertical Federated Learning without Third-Party Coordinator

Federated Learning is a new distributed learning mechanism which allows ...

FedPara: Low-rank Hadamard Product Parameterization for Efficient Federated Learning

To overcome the burdens on frequent model uploads and downloads during f...

Revealing and Protecting Labels in Distributed Training

Distributed learning paradigms such as federated learning often involve ...