DeepAI AI Chat
Log In Sign Up

Think Locally, Act Globally: Federated Learning with Local and Global Representations

01/06/2020
by   Paul Pu Liang, et al.
Carnegie Mellon University
6

Federated learning is an emerging research paradigm to train models on private data distributed over multiple devices. A key challenge involves keeping private all the data on each device and training a global model only by communicating parameters and updates. Overcoming this problem relies on the global model being sufficiently compact so that the parameters can be efficiently sent over communication channels such as wireless internet. Given the recent trend towards building deeper and larger neural networks, deploying such models in federated settings on real-world tasks is becoming increasingly difficult. To this end, we propose to augment federated learning with local representation learning on each device to learn useful and compact features from raw data. As a result, the global model can be smaller since it only operates on higher-level local representations. We show that our proposed method achieves superior or competitive results when compared to traditional federated approaches on a suite of publicly available real-world datasets spanning image recognition (MNIST, CIFAR) and multimodal learning (VQA). Our choice of local representation learning also reduces the number of parameters and updates that need to be communicated to and from the global model, thereby reducing the bottleneck in terms of communication cost. Finally, we show that our local models provide flexibility in dealing with online heterogeneous data and can be easily modified to learn fair representations that obfuscate protected attributes such as race, age, and gender, a feature crucial to preserving the privacy of on-device data.

READ FULL TEXT

page 2

page 18

12/10/2021

Federated Two-stage Learning with Sign-based Voting

Federated learning is a distributed machine learning mechanism where loc...
08/13/2020

WAFFLe: Weight Anonymized Factorization for Federated Learning

In domains where data are sensitive or private, there is great value in ...
02/10/2022

Personalization Improves Privacy-Accuracy Tradeoffs in Federated Optimization

Large-scale machine learning systems often involve data distributed acro...
10/17/2019

Overcoming Forgetting in Federated Learning on Non-IID Data

We tackle the problem of Federated Learning in the non i.i.d. case, in w...
10/05/2021

Securing Federated Learning: A Covert Communication-based Approach

Federated Learning Networks (FLNs) have been envisaged as a promising pa...
08/26/2022

Abnormal Local Clustering in Federated Learning

Federated learning is a model for privacy without revealing private data...
12/10/2020

Analysis and Optimal Edge Assignment For Hierarchical Federated Learning on Non-IID Data

Distributed learning algorithms aim to leverage distributed and diverse ...

Code Repositories

LG-FedAvg

[NeurIPS 2019 FL workshop] Federated Learning with Local and Global Representations


view repo