Adaptive Distillation for Decentralized Learning from Heterogeneous Clients

08/18/2020
by   Jiaxin Ma, et al.
8

This paper addresses the problem of decentralized learning to achieve a high-performance global model by asking a group of clients to share local models pre-trained with their own data resources. We are particularly interested in a specific case where both the client model architectures and data distributions are diverse, which makes it nontrivial to adopt conventional approaches such as Federated Learning and network co-distillation. To this end, we propose a new decentralized learning method called Decentralized Learning via Adaptive Distillation (DLAD). Given a collection of client models and a large number of unlabeled distillation samples, the proposed DLAD 1) aggregates the outputs of the client models while adaptively emphasizing those with higher confidence in given distillation samples and 2) trains the global model to imitate the aggregated outputs. Our extensive experimental evaluation on multiple public datasets (MNIST, CIFAR-10, and CINIC-10) demonstrates the effectiveness of the proposed method.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

02/16/2022

No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices

Federated learning (FL) is an important paradigm for training global mod...
08/30/2021

FedKD: Communication Efficient Federated Learning via Knowledge Distillation

Federated learning is widely used to learn intelligent models from decen...
05/31/2021

Unifying Distillation with Personalization in Federated Learning

Federated learning (FL) is a decentralized privacy-preserving learning t...
08/31/2021

GRP-FED: Addressing Client Imbalance in Federated Learning via Global-Regularized Personalization

Since data is presented long-tailed in reality, it is challenging for Fe...
12/02/2021

FedRAD: Federated Robust Adaptive Distillation

The robustness of federated learning (FL) is vital for the distributed t...
03/11/2022

Deep Class Incremental Learning from Decentralized Data

In this paper, we focus on a new and challenging decentralized machine l...
05/23/2019

Decentralized Learning of Generative Adversarial Networks from Multi-Client Non-iid Data

This work addresses a new problem of learning generative adversarial net...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.