DeepAI
Log In Sign Up

Enabling On-Device Training of Speech Recognition Models with Federated Dropout

10/07/2021
by   Dhruv Guliani, et al.
0

Federated learning can be used to train machine learning models on the edge on local data that never leave devices, providing privacy by default. This presents a challenge pertaining to the communication and computation costs associated with clients' devices. These costs are strongly correlated with the size of the model being trained, and are significant for state-of-the-art automatic speech recognition models. We propose using federated dropout to reduce the size of client models while training a full-size model server-side. We provide empirical evidence of the effectiveness of federated dropout, and propose a novel approach to vary the dropout rate applied at each layer. Furthermore, we find that federated dropout enables a set of smaller sub-models within the larger model to independently have low word error rates, making it easier to dynamically adjust the size of the model deployed for inference.

READ FULL TEXT

page 1

page 2

page 3

page 4

11/08/2020

Adaptive Federated Dropout: Improving Communication Efficiency and Generalization for Federated Learning

With more regulations tackling users' privacy-sensitive data protection ...
08/30/2022

Reducing Impacts of System Heterogeneity in Federated Learning using Weight Update Magnitudes

The widespread adoption of handheld devices have fueled rapid growth in ...
10/08/2021

Exploring Heterogeneous Characteristics of Layers in ASR Models for More Efficient Training

Transformer-based architectures have been the subject of research aimed ...
04/05/2021

Dynamic Encoder Transducer: A Flexible Solution For Trading Off Accuracy For Latency

We propose a dynamic encoder transducer (DET) for on-device speech recog...
11/20/2018

WEST: Word Encoded Sequence Transducers

Most of the parameters in large vocabulary models are used in embedding ...
01/26/2022

Fast Server Learning Rate Tuning for Coded Federated Dropout

In cross-device Federated Learning (FL), clients with low computational ...
11/09/2017

Analysis of Dropout in Online Learning

Deep learning is the state-of-the-art in fields such as visual object re...