Federated Learning with Intermediate Representation Regularization

10/28/2022
by   Ye Lin Tun, et al.
0

In contrast to centralized model training that involves data collection, federated learning (FL) enables remote clients to collaboratively train a model without exposing their private data. However, model performance usually degrades in FL due to the heterogeneous data generated by clients of diverse characteristics. One promising strategy to maintain good performance is by limiting the local training from drifting far away from the global model. Previous studies accomplish this by regularizing the distance between the representations learned by the local and global models. However, they only consider representations from the early layers of a model or the layer preceding the output layer. In this study, we introduce FedIntR, which provides a more fine-grained regularization by integrating the representations of intermediate layers into the local training process. Specifically, FedIntR computes a regularization term that encourages the closeness between the intermediate layer representations of the local and global models. Additionally, FedIntR automatically determines the contribution of each layer's representation to the regularization term based on the similarity between local and global representations. We conduct extensive experiments on various datasets to show that FedIntR can achieve equivalent or higher performance compared to the state-of-the-art approaches.

READ FULL TEXT

page 1

page 5

research
08/16/2022

FedMR: Fedreated Learning via Model Recombination

As a promising privacy-preserving machine learning method, Federated Lea...
research
04/21/2021

Covert Channel Attack to Federated Learning Systems

Federated learning (FL) goes beyond traditional, centralized machine lea...
research
03/04/2023

Federated Virtual Learning on Heterogeneous Data with Local-global Distillation

Despite Federated Learning (FL)'s trend for learning machine learning mo...
research
04/03/2023

FedIN: Federated Intermediate Layers Learning for Model Heterogeneity

Federated learning (FL) facilitates edge devices to cooperatively train ...
research
06/22/2021

Enabling Long-Term Cooperation in Cross-Silo Federated Learning: A Repeated Game Perspective

Cross-silo federated learning (FL) is a distributed learning approach wh...
research
04/08/2022

CD^2-pFed: Cyclic Distillation-guided Channel Decoupling for Model Personalization in Federated Learning

Federated learning (FL) is a distributed learning paradigm that enables ...
research
01/13/2023

Contrast with Major Classifier Vectors for Federated Medical Relation Extraction with Heterogeneous Label Distribution

Federated medical relation extraction enables multiple clients to train ...

Please sign up or login with your details

Forgot password? Click here to reset