Towards Model Agnostic Federated Learning Using Knowledge Distillation

10/28/2021
by   Andrei Afonin, et al.
0

An often unquestioned assumption underlying most current federated learning algorithms is that all the participants use identical model architectures. In this work, we initiate a theoretical study of model agnostic communication protocols which would allow data holders (agents) using different models to collaborate with each other and perform federated learning. We focus on the setting where the two agents are attempting to perform kernel regression using different kernels (and hence have different models). Our study yields a surprising result – the most natural algorithm of using alternating knowledge distillation (AKD) imposes overly strong regularization and may lead to severe under-fitting. Our theory also shows an interesting connection between AKD and the alternating projection algorithm for finding intersection of sets. Leveraging this connection, we propose a new algorithms which improve upon AKD. Our theoretical predictions also closely match real world experiments using neural networks. Thus, our work proposes a rich yet tractable framework for analyzing and developing new practical model agnostic federated learning algorithms.

READ FULL TEXT

Authors

page 1

page 2

page 3

page 4

05/02/2022

FedDKD: Federated Learning with Decentralized Knowledge Distillation

The performance of federated learning in neural networks is generally in...
10/08/2019

FedMD: Heterogenous Federated Learning via Model Distillation

Federated learning enables the creation of a powerful centralized model ...
08/30/2021

FedKD: Communication Efficient Federated Learning via Knowledge Distillation

Federated learning is widely used to learn intelligent models from decen...
05/03/2022

Efficient and Convergent Federated Learning

Federated learning has shown its advances over the last few years but is...
12/01/2020

Communication-Efficient Federated Distillation

Communication constraints are one of the major challenges preventing the...
09/04/2020

FedDistill: Making Bayesian Model Ensemble Applicable to Federated Learning

Federated learning aims to leverage users' own data and computational re...
02/25/2020

Three Approaches for Personalization with Applications to Federated Learning

The standard objective in machine learning is to train a single model fo...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.