Efficient and Reliable Overlay Networks for Decentralized Federated Learning

12/12/2021
by   Yifan Hua, et al.
0

We propose near-optimal overlay networks based on d-regular expander graphs to accelerate decentralized federated learning (DFL) and improve its generalization. In DFL a massive number of clients are connected by an overlay network, and they solve machine learning problems collaboratively without sharing raw data. Our overlay network design integrates spectral graph theory and the theoretical convergence and generalization bounds for DFL. As such, our proposed overlay networks accelerate convergence, improve generalization, and enhance robustness to clients failures in DFL with theoretical guarantees. Also, we present an efficient algorithm to convert a given graph to a practical overlay network and maintaining the network topology after potential client failures. We numerically verify the advantages of DFL with our proposed networks on various benchmark tasks, ranging from image classification to language modeling using hundreds of clients.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

11/01/2021

Implicit Model Specialization through DAG-based Decentralized Federated Learning

Federated learning allows a group of distributed clients to train a comm...
08/19/2021

Cross-Silo Federated Learning for Multi-Tier Networks with Vertical and Horizontal Data Partitioning

We consider federated learning in tiered communication networks. Our net...
07/30/2020

Communication-Efficient Federated Learning via Optimal Client Sampling

Federated learning is a private and efficient framework for learning mod...
12/24/2020

Decentralized Federated Learning via Mutual Knowledge Transfer

In this paper, we investigate the problem of decentralized federated lea...
10/26/2021

Fault-Tolerant Federated Reinforcement Learning with Theoretical Guarantee

The growing literature of Federated Learning (FL) has recently inspired ...
09/10/2019

The Evolution of Decentralized ICT Networks

Traditional networking systems, especially the famous 3 tier topology de...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.