Unifying Distillation with Personalization in Federated Learning

05/31/2021
by   Siddharth Divi, et al.
0

Federated learning (FL) is a decentralized privacy-preserving learning technique in which clients learn a joint collaborative model through a central aggregator without sharing their data. In this setting, all clients learn a single common predictor (FedAvg), which does not generalize well on each client's local data due to the statistical data heterogeneity among clients. In this paper, we address this problem with PersFL, a discrete two-stage personalized learning algorithm. In the first stage, PersFL finds the optimal teacher model of each client during the FL training phase. In the second stage, PersFL distills the useful knowledge from optimal teachers into each user's local model. The teacher model provides each client with some rich, high-level representation that a client can easily adapt to its local model, which overcomes the statistical heterogeneity present at different clients. We evaluate PersFL on CIFAR-10 and MNIST datasets using three data-splitting strategies to control the diversity between clients' data distributions. We empirically show that PersFL outperforms FedAvg and three state-of-the-art personalization methods, pFedMe, Per-FedAvg, and FedPer on majority data-splits with minimal communication cost. Further, we study the performance of PersFL on different distillation objectives, how this performance is affected by the equitable notion of fairness among clients, and the number of required communication rounds. PersFL code is available at https://tinyurl.com/hdh5zhxs for public use and validation.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

07/28/2021

New Metrics to Evaluate the Performance and Fairness of Personalized Federated Learning

In Federated Learning (FL), the clients learn a single global model (Fed...
02/26/2021

FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout

Federated Learning (FL) has been gaining significant traction across dif...
08/30/2021

FedKD: Communication Efficient Federated Learning via Knowledge Distillation

Federated learning is widely used to learn intelligent models from decen...
09/04/2009

Assessing the Impact of Informedness on a Consultant's Profit

We study the notion of informedness in a client-consultant setting. Usin...
08/18/2020

Adaptive Distillation for Decentralized Learning from Heterogeneous Clients

This paper addresses the problem of decentralized learning to achieve a ...
05/02/2021

Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity

The traditional approach in FL tries to learn a single global model coll...
06/29/2021

Personalized Federated Learning with Gaussian Processes

Federated learning aims to learn a global model that performs well on cl...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.