Generalized Federated Learning via Sharpness Aware Minimization

06/06/2022
by   Zhe Qu, et al.
0

Federated Learning (FL) is a promising framework for performing privacy-preserving, distributed learning with a set of clients. However, the data distribution among clients often exhibits non-IID, i.e., distribution shift, which makes efficient optimization difficult. To tackle this problem, many FL algorithms focus on mitigating the effects of data heterogeneity across clients by increasing the performance of the global model. However, almost all algorithms leverage Empirical Risk Minimization (ERM) to be the local optimizer, which is easy to make the global model fall into a sharp valley and increase a large deviation of parts of local clients. Therefore, in this paper, we revisit the solutions to the distribution shift problem in FL with a focus on local learning generality. To this end, we propose a general, effective algorithm, , based on Sharpness Aware Minimization (SAM) local optimizer, and develop a momentum FL algorithm to bridge local and global models, . Theoretically, we show the convergence analysis of these two algorithms and demonstrate the generalization bound of . Empirically, our proposed algorithms substantially outperform existing FL studies and significantly decrease the learning deviation.

READ FULL TEXT

page 1

page 2

page 3

page 4

04/09/2022

Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning

Federated learning (FL) is a promising strategy for performing privacy-p...
08/22/2021

Flexible Clustered Federated Learning for Client-Level Data Distribution Shift

Federated Learning (FL) enables the multiple participating devices to co...
05/23/2022

FL Games: A federated learning framework for distribution shifts

Federated learning aims to train predictive models for data that is dist...
12/09/2021

Specificity-Preserving Federated Learning for MR Image Reconstruction

Federated learning (FL) can be used to improve data privacy and efficien...
03/28/2022

Federated Learning with Position-Aware Neurons

Federated Learning (FL) fuses collaborative models from local nodes with...
12/28/2021

Robust Convergence in Federated Learning through Label-wise Clustering

Non-IID dataset and heterogeneous environment of the local clients are r...
07/26/2021

Aggregate or Not? Exploring Where to Privatize in DNN Based Federated Learning Under Different Non-IID Scenes

Although federated learning (FL) has recently been proposed for efficien...