DeepAI AI Chat
Log In Sign Up

Differentially Private Variational Dropout

by   Beyza Ermis, et al.
Boğaziçi University

Deep neural networks with their large number of parameters are highly flexible learning systems. The high flexibility in such networks brings with some serious problems such as overfitting, and regularization is used to address this problem. A currently popular and effective regularization technique for controlling the overfitting is dropout. Often, large data collections required for neural networks contain sensitive information such as the medical histories of patients, and the privacy of the training data should be protected. In this paper, we modify the recently proposed variational dropout technique which provided an elegant Bayesian interpretation to dropout, and show that the intrinsic noise in the variational dropout can be exploited to obtain a degree of differential privacy. The iterative nature of training neural networks presents a challenge for privacy-preserving estimation since multiple iterations increase the amount of noise added. We overcome this by using a relaxed notion of differential privacy, called concentrated differential privacy, which provides tighter estimates on the overall privacy loss. We demonstrate the accuracy of our privacy-preserving variational dropout algorithm on benchmark datasets.


Differentially Private Dropout

Large data collections required for the training of neural networks ofte...

To Drop or Not to Drop: Robustness, Consistency and Differential Privacy Properties of Dropout

Training deep belief networks (DBNs) requires optimizing a non-convex fu...

Variational Bayes In Private Settings (VIPS)

We provide a general framework for privacy-preserving variational Bayes ...

Privacy-Preserving Distributed Deep Learning for Clinical Data

Deep learning with medical data often requires larger samples sizes than...

Private Topic Modeling

We develop a privatised stochastic variational inference method for Late...

A note on privacy preserving iteratively reweighted least squares

Iteratively reweighted least squares (IRLS) is a widely-used method in m...