Decentralized Federated Learning via Mutual Knowledge Transfer

12/24/2020
by   Chengxi Li, et al.
0

In this paper, we investigate the problem of decentralized federated learning (DFL) in Internet of things (IoT) systems, where a number of IoT clients train models collectively for a common task without sharing their private training data in the absence of a central server. Most of the existing DFL schemes are composed of two alternating steps, i.e., gradient update and model averaging. However, averaging of model parameters directly to fuse different models at the local clients suffers from client-drift in the local updates especially when the training data are heterogeneous across different clients. This leads to slow convergence and degraded learning performance. As a possible solution, we propose the decentralized federated learning via mutual knowledge transfer (Def-KT) algorithm where local clients fuse models by transferring their learnt knowledge to each other. Our experiments on the MNIST, Fashion-MNIST, and CIFAR10 datasets reveal that the proposed Def-KT algorithm significantly outperforms the baseline DFL methods with model averaging, i.e., Combo and FullAvg, especially when the training data are not independent and identically distributed (non-IID) across different clients.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

11/01/2021

Implicit Model Specialization through DAG-based Decentralized Federated Learning

Federated learning allows a group of distributed clients to train a comm...
10/14/2019

SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning

Federated learning is a key scenario in modern large-scale machine learn...
08/07/2021

The Effect of Training Parameters and Mechanisms on Decentralized Federated Learning based on MNIST Dataset

Federated Learning is an algorithm suited for training models on decentr...
07/20/2021

Precision-Weighted Federated Learning

Federated Learning using the Federated Averaging algorithm has shown gre...
02/06/2021

Multi-Tier Federated Learning for Vertically Partitioned Data

We consider decentralized model training in tiered communication network...
03/30/2022

Perfectly Accurate Membership Inference by a Dishonest Central Server in Federated Learning

Federated Learning is expected to provide strong privacy guarantees, as ...
12/12/2021

Efficient and Reliable Overlay Networks for Decentralized Federated Learning

We propose near-optimal overlay networks based on d-regular expander gra...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.