Differentially Private Distributed Learning for Language Modeling Tasks

12/20/2017
by   Vadim Popov, et al.
0

One of the big challenges in machine learning applications is that training data can be different from the real-world data faced by the algorithm. In language modeling, the language of users (e.g. in private messaging) could change in a year and be completely different from what we observe in publicly available data. At the same time, public data can be used for obtaining general knowledge (i.e. general model of English). We study approaches to distributed fine-tuning of a general model on user private data with the additional requirement of maintaining the quality on the general data. Our experiments demonstrate that a technique based on model averaging and random rehearsal outperforms an approach based on transfer learning, and show that the proposed method improves prediction quality in a reasonable time. The procedure leads to an almost 70 keystroke saving rate on informal English texts compared to a basic model trained on Wikipedia. We also propose an experimental framework for evaluating differential privacy of distributed training of language models and show that our approach has good privacy guarantees.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/13/2020

Differentially Private Language Models Benefit from Public Pre-training

Language modeling is a keystone task in natural language processing. Whe...
research
10/25/2022

Differentially Private Language Models for Secure Data Sharing

To protect the privacy of individuals whose data is being shared, it is ...
research
08/30/2021

Selective Differential Privacy for Language Modeling

With the increasing adoption of language models in applications involvin...
research
05/20/2023

Can Public Large Language Models Help Private Cross-device Federated Learning?

We study (differentially) private federated learning (FL) of language mo...
research
05/24/2022

CryptoTL: Private, efficient and secure transfer learning

Big data has been a pervasive catchphrase in recent years, but dealing w...
research
09/12/2019

Differentially Private Meta-Learning

Parameter-transfer is a well-known and versatile approach for meta-learn...
research
09/21/2020

Training Production Language Models without Memorizing User Data

This paper presents the first consumer-scale next-word prediction (NWP) ...

Please sign up or login with your details

Forgot password? Click here to reset