Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints

10/04/2019
by   Felix Sattler, et al.
42

Federated Learning (FL) is currently the most widely adopted framework for collaborative training of (deep) machine learning models under privacy constraints. Albeit it's popularity, it has been observed that Federated Learning yields suboptimal results if the local clients' data distributions diverge. To address this issue, we present Clustered Federated Learning (CFL), a novel Federated Multi-Task Learning (FMTL) framework, which exploits geometric properties of the FL loss surface, to group the client population into clusters with jointly trainable data distributions. In contrast to existing FMTL approaches, CFL does not require any modifications to the FL communication protocol to be made, is applicable to general non-convex objectives (in particular deep neural networks) and comes with strong mathematical guarantees on the clustering quality. CFL is flexible enough to handle client populations that vary over time and can be implemented in a privacy preserving way. As clustering is only performed after Federated Learning has converged to a stationary point, CFL can be viewed as a post-processing method that will always achieve greater or equal performance than conventional FL by allowing clients to arrive at more specialized models. We verify our theoretical analysis in experiments with deep convolutional and recurrent neural networks on commonly used Federated Learning datasets.

READ FULL TEXT
02/13/2022

On the Convergence of Clustered Federated Learning

In a federated learning system, the clients, e.g. mobile devices and org...
08/23/2021

Federated Multi-Task Learning under a Mixture of Distributions

The increasing size of data generated by smartphones and IoT devices mot...
08/22/2021

Personalised Federated Learning: A Combinational Approach

Federated learning (FL) is a distributed machine learning approach invol...
05/07/2021

Towards Practical Watermark for Deep Neural Networks in Federated Learning

With the wide application of deep neural networks, it is important to ve...
03/21/2020

Dynamic Sampling and Selective Masking for Communication-Efficient Federated Learning

Federated learning (FL) is a novel machine learning setting which enable...
04/07/2022

Multi-Task Distributed Learning using Vision Transformer with Random Patch Permutation

The widespread application of artificial intelligence in health research...
05/12/2021

Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learning

This work addresses the problem of optimizing communications between ser...

Code Repositories

clustered-federated-learning

Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints


view repo