Federated learning with hierarchical clustering of local updates to improve training on non-IID data

04/24/2020
by   Christopher Briggs, et al.
0

Federated learning (FL) is a well established method for performing machine learning tasks over massively distributed data. However in settings where data is distributed in a non-iid (not independent and identically distributed) fashion – as is typical in real world situations – the joint model produced by FL suffers in terms of test set accuracy and/or communication costs compared to training on iid data. We show that learning a single joint model is often not optimal in the presence of certain types of non-iid data. In this work we present a modification to FL by introducing a hierarchical clustering step (FL+HC) to separate clusters of clients by the similarity of their local updates to the global joint model. Once separated, the clusters are trained independently and in parallel on specialised models. We present a robust empirical analysis of the hyperparameters for FL+HC for several iid and non-iid settings. We show how FL+HC allows model training to converge in fewer communication rounds (significantly so under some non-iid settings) compared to FL without clustering. Additionally, FL+HC allows for a greater percentage of clients to reach a target accuracy compared to standard FL. Finally we make suggestions for good default hyperparameters to promote superior performing specialised models without modifying the the underlying federated learning communication protocol.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/28/2020

Semi-Federated Learning

Federated learning (FL) enables massive distributed Information and Comm...
research
05/02/2023

FedAVO: Improving Communication Efficiency in Federated Learning with African Vultures Optimizer

Federated Learning (FL), a distributed machine learning technique has re...
research
05/08/2023

FedHB: Hierarchical Bayesian Federated Learning

We propose a novel hierarchical Bayesian approach to Federated Learning ...
research
04/27/2022

FedShuffle: Recipes for Better Use of Local Work in Federated Learning

The practice of applying several local updates before aggregation across...
research
08/16/2019

Federated Learning with Additional Mechanisms on Clients to Reduce Communication Costs

Federated learning (FL) enables on-device training over distributed netw...
research
01/18/2022

Towards Federated Clustering: A Federated Fuzzy c-Means Algorithm (FFCM)

Federated Learning (FL) is a setting where multiple parties with distrib...
research
01/31/2022

Federated Learning with Erroneous Communication Links

In this paper, we consider the federated learning (FL) problem in the pr...

Please sign up or login with your details

Forgot password? Click here to reset