PFL-MoE: Personalized Federated Learning Based on Mixture of Experts

12/31/2020
by   Binbin Guo, et al.
0

Federated learning (FL) is an emerging distributed machine learning paradigm that avoids data sharing among training nodes so as to protect data privacy. Under coordination of the FL server, each client conducts model training using its own computing resource and private data set. The global model can be created by aggregating the training results of clients. To cope with highly non-IID data distributions, personalized federated learning (PFL) has been proposed to improve overall performance by allowing each client to learn a personalized model. However, one major drawback of a personalized model is the loss of generalization. To achieve model personalization while maintaining generalization, in this paper, we propose a new approach, named PFL-MoE, which mixes outputs of the personalized model and global model via the MoE architecture. PFL-MoE is a generic approach and can be instantiated by integrating existing PFL algorithms. Particularly, we propose the PFL-MF algorithm which is an instance of PFL-MoE based on the freeze-base PFL algorithm. We further improve PFL-MF by enhancing the decision-making ability of MoE gating network and propose a variant algorithm PFL-MFE. We demonstrate the effectiveness of PFL-MoE by training the LeNet-5 and VGG-16 models on the Fashion-MNIST and CIFAR-10 datasets with non-IID partitions.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/16/2020

Personalized Federated Learning with Moreau Envelopes

Federated learning (FL) is a decentralized and privacy-preserving machin...
research
08/04/2021

Personalized Federated Learning with Clustering: Non-IID Heart Rate Variability Data Application

While machine learning techniques are being applied to various fields fo...
research
06/03/2023

Forgettable Federated Linear Learning with Certified Data Removal

Federated learning (FL) is a trending distributed learning framework tha...
research
12/02/2021

Personalized Federated Learning of Driver Prediction Models for Autonomous Driving

Autonomous vehicles (AVs) must interact with a diverse set of human driv...
research
08/07/2020

LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasets

Federated learning is a popular distributed machine learning paradigm wi...
research
11/03/2022

FedTP: Federated Learning by Transformer Personalization

Federated learning is an emerging learning paradigm where multiple clien...
research
07/02/2021

On Bridging Generic and Personalized Federated Learning

Federated learning is promising for its ability to collaboratively train...

Please sign up or login with your details

Forgot password? Click here to reset