Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data

09/14/2020
by   Qianqian Tong, et al.
0

Federated learning allows loads of edge computing devices to collaboratively learn a global model without data sharing. The analysis with partial device participation under non-IID and unbalanced data reflects more reality. In this work, we propose federated learning versions of adaptive gradient methods - Federated AGMs - which employ both the first-order and second-order momenta, to alleviate generalization performance deterioration caused by dissimilarity of data population among devices. To further improve the test performance, we compare several schemes of calibration for the adaptive learning rate, including the standard Adam calibrated by ϵ, p-Adam, and one calibrated by an activation function. Our analysis provides the first set of theoretical results that the proposed (calibrated) Federated AGMs converge to a first-order stationary point under non-IID and unbalanced data settings for nonconvex optimization. We perform extensive experiments to compare these federated learning methods with the state-of-the-art FedAvg, FedMomentum and SCAFFOLD and to assess the different calibration schemes and the advantages of AGMs over the current federated learning methods.

READ FULL TEXT
research
02/29/2020

Adaptive Federated Optimization

Federated learning is a distributed machine learning paradigm in which a...
research
07/04/2019

On the Convergence of FedAvg on Non-IID Data

Federated learning enables a large amount of edge computing devices to l...
research
12/02/2020

Second-Order Guarantees in Federated Learning

Federated learning is a useful framework for centralized learning from d...
research
02/12/2020

Towards Federated Learning: Robustness Analytics to Data Heterogeneity

Federated Learning allows remote centralized server training models with...
research
09/18/2023

FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data

Federated learning is an emerging distributed machine learning method, e...
research
06/20/2020

FedMGDA+: Federated Learning meets Multi-objective Optimization

Federated learning has emerged as a promising, massively distributed way...
research
12/14/2020

Federated Learning under Importance Sampling

Federated learning encapsulates distributed learning strategies that are...

Please sign up or login with your details

Forgot password? Click here to reset