Improving the Model Consistency of Decentralized Federated Learning

02/08/2023
by   Yifan Shi, et al.
10

To mitigate the privacy leakages and communication burdens of Federated Learning (FL), decentralized FL (DFL) discards the central server and each client only communicates with its neighbors in a decentralized communication network. However, existing DFL suffers from high inconsistency among local clients, which results in severe distribution shift and inferior performance compared with centralized FL (CFL), especially on heterogeneous data or sparse communication topology. To alleviate this issue, we propose two DFL algorithms named DFedSAM and DFedSAM-MGS to improve the performance of DFL. Specifically, DFedSAM leverages gradient perturbation to generate local flat models via Sharpness Aware Minimization (SAM), which searches for models with uniformly low loss values. DFedSAM-MGS further boosts DFedSAM by adopting Multiple Gossip Steps (MGS) for better model consistency, which accelerates the aggregation of local flat models and better balances communication complexity and generalization. Theoretically, we present improved convergence rates 𝒪(1/√(KT)+1/T+1/K^1/2T^3/2(1-λ)^2) and 𝒪(1/√(KT)+1/T+λ^Q+1/K^1/2T^3/2(1-λ^Q)^2) in non-convex setting for DFedSAM and DFedSAM-MGS, respectively, where 1-λ is the spectral gap of gossip matrix and Q is the number of MGS. Empirically, our methods can achieve competitive performance compared with CFL methods and outperform existing DFL methods.

READ FULL TEXT
research
08/16/2023

DFedADMM: Dual Constraints Controlled Model Inconsistency for Decentralized Federated Learning

To address the communication burden issues associated with federated lea...
research
05/24/2023

Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Training

Personalized federated learning (PFL) aims to produce the greatest perso...
research
05/26/2020

Continual Local Training for Better Initialization of Federated Models

Federated learning (FL) refers to the learning paradigm that trains mach...
research
12/20/2021

HarmoFL: Harmonizing Local and Global Drifts in Federated Learning on Heterogeneous Medical Images

Multiple medical institutions collaboratively training a model using fed...
research
05/01/2023

Towards the Flatter Landscape and Better Generalization in Federated Learning under Client-level Differential Privacy

To defend the inference attacks and mitigate the sensitive information l...
research
11/10/2021

DACFL: Dynamic Average Consensus Based Federated Learning in Decentralized Topology

Federated learning (FL) is a burgeoning distributed machine learning fra...
research
05/19/2023

Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscape

In federated learning (FL), a cluster of local clients are chaired under...

Please sign up or login with your details

Forgot password? Click here to reset