Federated Learning Of Out-Of-Vocabulary Words

by   Mingqing Chen, et al.

We demonstrate that a character-level recurrent neural network is able to learn out-of-vocabulary (OOV) words under federated learning settings, for the purpose of expanding the vocabulary of a virtual keyboard for smartphones without exporting sensitive text to servers. High-frequency words can be sampled from the trained generative model by drawing from the joint posterior directly. We study the feasibility of the approach in two settings: (1) using simulated federated learning on a publicly available non-IID per-user dataset from a popular social networking website, (2) using federated learning on data hosted on user mobile devices. The model achieves good recall and precision compared to ground-truth OOV words in setting (1). With (2) we demonstrate the practicality of this approach by showing that we can learn meaningful OOV words with good character-level prediction accuracy and cross entropy loss.


page 1

page 2

page 3

page 4


Federated Learning for Mobile Keyboard Prediction

We train a recurrent neural network language model using a distributed, ...

Federated Learning for Emoji Prediction in a Mobile Keyboard

We show that a word-level recurrent neural network can predict emoji fro...

Training a Tokenizer for Free with Private Federated Learning

Federated learning with differential privacy, i.e. private federated lea...

Bayesian Nonparametric Federated Learning of Neural Networks

In federated learning problems, data is scattered across different serve...

Federated Learning of N-gram Language Models

We propose algorithms to train production-quality n-gram language models...

Fed-Focal Loss for imbalanced data classification in Federated Learning

The Federated Learning setting has a central server coordinating the tra...

Federated Evaluation of On-device Personalization

Federated learning is a distributed, on-device computation framework tha...

Please sign up or login with your details

Forgot password? Click here to reset