Stragglers Are Not Disaster: A Hybrid Federated Learning Algorithm with Delayed Gradients

02/12/2021
by   Xingyu Li, et al.
0

Federated learning (FL) is a new machine learning framework which trains a joint model across a large amount of decentralized computing devices. Existing methods, e.g., Federated Averaging (FedAvg), are able to provide an optimization guarantee by synchronously training the joint model, but usually suffer from stragglers, i.e., IoT devices with low computing power or communication bandwidth, especially on heterogeneous optimization problems. To mitigate the influence of stragglers, this paper presents a novel FL algorithm, namely Hybrid Federated Learning (HFL), to achieve a learning balance in efficiency and effectiveness. It consists of two major components: synchronous kernel and asynchronous updater. Unlike traditional synchronous FL methods, our HFL introduces the asynchronous updater which actively pulls unsynchronized and delayed local weights from stragglers. An adaptive approximation method, Adaptive Delayed-SGD (AD-SGD), is proposed to merge the delayed local updates into the joint model. The theoretical analysis of HFL shows that the convergence rate of the proposed algorithm is 𝒪(1/t+τ) for both convex and non-convex optimization problems.

READ FULL TEXT

page 9

page 19

page 20

page 21

06/11/2021

Federated Learning with Buffered Asynchronous Aggregation

Federated Learning (FL) trains a shared model across distributed devices...
05/27/2022

AsyncFedED: Asynchronous Federated Learning with Euclidean Distance based Adaptive Weight Aggregation

In an asynchronous federated learning framework, the server updates the ...
03/10/2019

Asynchronous Federated Optimization

Federated learning enables training on a massive number of edge devices....
06/02/2022

Resource Allocation for Compression-aided Federated Learning with High Distortion Rate

Recently, a considerable amount of works have been made to tackle the co...
06/29/2022

AFAFed – Protocol analysis

In this paper, we design, analyze the convergence properties and address...
06/10/2022

On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyond

The FedProx algorithm is a simple yet powerful distributed proximal poin...
03/02/2022

Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Data

Federated learning (FL) is an emerging privacy-preserving paradigm that ...