Federated Multiple Label Hashing (FedMLH): Communication Efficient Federated Learning on Extreme Classification Tasks

by   Zhenwei Dai, et al.
Rice University

Federated learning enables many local devices to train a deep learning model jointly without sharing the local data. Currently, most of federated training schemes learns a global model by averaging the parameters of local models. However, most of these training schemes suffer from high communication cost resulted from transmitting full local model parameters. Moreover, directly averaging model parameters leads to a significant performance degradation, due to the class-imbalanced non-iid data on different devices. Especially for the real life federated learning tasks involving extreme classification, (1) communication becomes the main bottleneck since the model size increases proportionally to the number of output classes; (2) extreme classification (such as user recommendation) normally have extremely imbalanced classes and heterogeneous data on different devices. To overcome this problem, we propose federated multiple label hashing (FedMLH), which leverages label hashing to simultaneously reduce the model size (up to 3.40X decrease) with communication cost (up to 18.75X decrease) and achieves significant better accuracy (up to 35.5 increase) for free on the federated extreme classification tasks compared to federated average algorithm.


page 1

page 2

page 3

page 4


FedHeN: Federated Learning in Heterogeneous Networks

We propose a novel training recipe for federated learning with heterogen...

FedControl: When Control Theory Meets Federated Learning

To date, the most popular federated learning algorithms use coordinate-w...

Improving the Robustness of Federated Learning for Severely Imbalanced Datasets

With the ever increasing data deluge and the success of deep neural netw...

Extreme Classification in Log Memory

We present Merged-Averaged Classifiers via Hashing (MACH) for K-classifi...

Extreme Classification in Log Memory using Count-Min Sketch: A Case Study of Amazon Search with 50M Products

In the last decade, it has been shown that many hard AI tasks, especiall...

Federated Learning with Only Positive Labels

We consider learning a multi-class classification model in the federated...

Proof-of-Federated-Learning-Subchain: Free Partner Selection Subchain Based on Federated Learning

The continuous thriving of the Blockchain society motivates research in ...

Please sign up or login with your details

Forgot password? Click here to reset