Benchmarking Differential Privacy and Federated Learning for BERT Models

06/26/2021

∙

Natural Language Processing (NLP) techniques can be applied to help with the diagnosis of medical conditions such as depression, using a collection of a person's utterances. Depression is a serious medical illness that can have adverse effects on how one feels, thinks, and acts, which can lead to emotional and physical problems. Due to the sensitive nature of such data, privacy measures need to be taken for handling and training models with such data. In this work, we study the effects that the application of Differential Privacy (DP) has, in both a centralized and a Federated Learning (FL) setup, on training contextualized language models (BERT, ALBERT, RoBERTa and DistilBERT). We offer insights on how to privately train NLP models and what architectures and setups provide more desirable privacy utility trade-offs. We envisage this work to be used in future healthcare and mental health studies to keep medical history private. Therefore, we provide an open-source implementation of this work.

READ FULL TEXT

Benchmarking Differential Privacy and Federated Learning for BERT Models

Privacy enabled Financial Text Classification using Differential Privacy and Federated Learning

Interpretability of Fine-grained Classification of Sadness and Depression

Global Differential Privacy for Distributed Metaverse Healthcare Systems

Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Models

Multi-Site Clinical Federated Learning using Recursive and Attentive Models and NVFlare

How Much User Context Do We Need? Privacy by Design in Mental Health NLP Application

Federated Learning of Gboard Language Models with Differential Privacy

Benchmarking Differential Privacy and Federated Learning for BERT Models

Related Research

Privacy enabled Financial Text Classification using Differential Privacy and Federated Learning

Interpretability of Fine-grained Classification of Sadness and Depression

Global Differential Privacy for Distributed Metaverse Healthcare Systems

Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Models

Multi-Site Clinical Federated Learning using Recursive and Attentive Models and NVFlare

How Much User Context Do We Need? Privacy by Design in Mental Health NLP Application

Federated Learning of Gboard Language Models with Differential Privacy