Personalized Federated Learning with Gaussian Processes

06/29/2021
by   Idan Achituve, et al.
0

Federated learning aims to learn a global model that performs well on client devices with limited cross-client communication. Personalized federated learning (PFL) further extends this setup to handle data heterogeneity between clients by learning personalized models. A key challenge in this setting is to learn effectively across clients even though each client has unique data that is often limited in size. Here we present pFedGP, a solution to PFL that is based on Gaussian processes (GPs) with deep kernel learning. GPs are highly expressive models that work well in the low data regime due to their Bayesian nature. However, applying GPs to PFL raises multiple challenges. Mainly, GPs performance depends heavily on access to a good kernel function, and learning a kernel requires a large training set. Therefore, we propose learning a shared kernel function across all clients, parameterized by a neural network, with a personal GP classifier for each client. We further extend pFedGP to include inducing points using two novel methods, the first helps to improve generalization in the low data regime and the second reduces the computational cost. We derive a PAC-Bayes generalization bound on novel clients and empirically show that it gives non-vacuous guarantees. Extensive experiments on standard PFL benchmarks with CIFAR-10, CIFAR-100, and CINIC-10, and on a new setup of learning under input noise show that pFedGP achieves well-calibrated predictions while significantly outperforming baseline methods, reaching up to 21

READ FULL TEXT

Authors

page 9

page 22

03/08/2021

Personalized Federated Learning using Hypernetworks

Personalized federated learning is tasked with training machine learning...
06/13/2022

Federated Bayesian Neural Regression: A Scalable Global Federated Gaussian Process

In typical scenarios where the Federated Learning (FL) framework applies...
08/07/2020

LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasets

Federated learning is a popular distributed machine learning paradigm wi...
07/07/2020

Personalized Federated Learning: An Attentive Collaboration Approach

For the challenging computational environment of IOT/edge computing, per...
06/16/2022

Personalized Federated Learning via Variational Bayesian Inference

Federated learning faces huge challenges from model overfitting due to t...
09/16/2021

Subspace Learning for Personalized Federated Optimization

As data is generated and stored almost everywhere, learning a model from...
11/04/2021

Parameterized Knowledge Transfer for Personalized Federated Learning

In recent years, personalized federated learning (pFL) has attracted inc...

Code Repositories

pFedGP

Code for Personalized Federated Learning with Gaussian Processes


view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.