Log In Sign Up

Enabling On-Device Training of Speech Recognition Models with Federated Dropout

by   Dhruv Guliani, et al.

Federated learning can be used to train machine learning models on the edge on local data that never leave devices, providing privacy by default. This presents a challenge pertaining to the communication and computation costs associated with clients' devices. These costs are strongly correlated with the size of the model being trained, and are significant for state-of-the-art automatic speech recognition models. We propose using federated dropout to reduce the size of client models while training a full-size model server-side. We provide empirical evidence of the effectiveness of federated dropout, and propose a novel approach to vary the dropout rate applied at each layer. Furthermore, we find that federated dropout enables a set of smaller sub-models within the larger model to independently have low word error rates, making it easier to dynamically adjust the size of the model deployed for inference.


page 1

page 2

page 3

page 4


Adaptive Federated Dropout: Improving Communication Efficiency and Generalization for Federated Learning

With more regulations tackling users' privacy-sensitive data protection ...

Reducing Impacts of System Heterogeneity in Federated Learning using Weight Update Magnitudes

The widespread adoption of handheld devices have fueled rapid growth in ...

Exploring Heterogeneous Characteristics of Layers in ASR Models for More Efficient Training

Transformer-based architectures have been the subject of research aimed ...

Dynamic Encoder Transducer: A Flexible Solution For Trading Off Accuracy For Latency

We propose a dynamic encoder transducer (DET) for on-device speech recog...

WEST: Word Encoded Sequence Transducers

Most of the parameters in large vocabulary models are used in embedding ...

Fast Server Learning Rate Tuning for Coded Federated Dropout

In cross-device Federated Learning (FL), clients with low computational ...

Analysis of Dropout in Online Learning

Deep learning is the state-of-the-art in fields such as visual object re...