CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learning

06/14/2021
by   Haibo Yang, et al.
0

Federated learning (FL) is a prevailing distributed learning paradigm, where a large number of workers jointly learn a model without sharing their training data. However, high communication costs could arise in FL due to large-scale (deep) learning models and bandwidth-constrained connections. In this paper, we introduce a communication-efficient algorithmic framework called CFedAvg for FL with non-i.i.d. datasets, which works with general (biased or unbiased) SNR-constrained compressors. We analyze the convergence rate of CFedAvg for non-convex functions with constant and decaying learning rates. The CFedAvg algorithm can achieve an 𝒪(1 / √(mKT) + 1 / T) convergence rate with a constant learning rate, implying a linear speedup for convergence as the number of workers increases, where K is the number of local steps, T is the number of total communication rounds, and m is the total worker number. This matches the convergence rate of distributed/federated learning without compression, thus achieving high communication efficiency while not sacrificing learning accuracy in FL. Furthermore, we extend CFedAvg to cases with heterogeneous local steps, which allows different workers to perform a different number of local steps to better adapt to their own circumstances. The interesting observation in general is that the noise/variance introduced by compressors does not affect the overall convergence rate order for non-i.i.d. FL. We verify the effectiveness of our CFedAvg algorithm on three datasets with two gradient compression schemes of different compression ratios.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

01/27/2021

Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning

Federated learning (FL) is a distributed machine learning architecture t...
11/06/2018

Elastic CoCoA: Scaling In to Improve Convergence

In this paper we experimentally analyze the convergence behavior of CoCo...
08/23/2021

Anarchic Federated Learning

Present-day federated learning (FL) systems deployed over edge networks ...
12/13/2021

Optimal Rate Adaption in Federated Learning with Compressed Communications

Federated Learning (FL) incurs high communication overhead, which can be...
08/06/2019

Motivating Workers in Federated Learning: A Stackelberg Game Perspective

Due to the large size of the training data, distributed learning approac...
08/12/2021

Communication Optimization in Large Scale Federated Learning using Autoencoder Compressed Weight Updates

Federated Learning (FL) solves many of this decade's concerns regarding ...
06/25/2020

Artemis: tight convergence guarantees for bidirectional compression in Federated Learning

We introduce a new algorithm - Artemis - tackling the problem of learnin...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.