Think Locally, Act Globally: Federated Learning with Local and Global Representations

01/06/2020
by   Paul Pu Liang, et al.
6

Federated learning is an emerging research paradigm to train models on private data distributed over multiple devices. A key challenge involves keeping private all the data on each device and training a global model only by communicating parameters and updates. Overcoming this problem relies on the global model being sufficiently compact so that the parameters can be efficiently sent over communication channels such as wireless internet. Given the recent trend towards building deeper and larger neural networks, deploying such models in federated settings on real-world tasks is becoming increasingly difficult. To this end, we propose to augment federated learning with local representation learning on each device to learn useful and compact features from raw data. As a result, the global model can be smaller since it only operates on higher-level local representations. We show that our proposed method achieves superior or competitive results when compared to traditional federated approaches on a suite of publicly available real-world datasets spanning image recognition (MNIST, CIFAR) and multimodal learning (VQA). Our choice of local representation learning also reduces the number of parameters and updates that need to be communicated to and from the global model, thereby reducing the bottleneck in terms of communication cost. Finally, we show that our local models provide flexibility in dealing with online heterogeneous data and can be easily modified to learn fair representations that obfuscate protected attributes such as race, age, and gender, a feature crucial to preserving the privacy of on-device data.

READ FULL TEXT

page 2

page 18

research
12/10/2021

Federated Two-stage Learning with Sign-based Voting

Federated learning is a distributed machine learning mechanism where loc...
research
08/13/2020

WAFFLe: Weight Anonymized Factorization for Federated Learning

In domains where data are sensitive or private, there is great value in ...
research
02/10/2022

Personalization Improves Privacy-Accuracy Tradeoffs in Federated Optimization

Large-scale machine learning systems often involve data distributed acro...
research
02/10/2020

Federated Learning of a Mixture of Global and Local Models

We propose a new optimization formulation for training federated learnin...
research
10/17/2019

Overcoming Forgetting in Federated Learning on Non-IID Data

We tackle the problem of Federated Learning in the non i.i.d. case, in w...
research
02/15/2020

Federated Learning with Matched Averaging

Federated learning allows edge devices to collaboratively learn a shared...
research
08/02/2022

Mitigating Biases in Student Performance Prediction via Attention-Based Personalized Federated Learning

Traditional learning-based approaches to student modeling generalize poo...

Please sign up or login with your details

Forgot password? Click here to reset