DeepAI AI Chat
Log In Sign Up

Differentially private training of neural networks with Langevin dynamics for calibrated predictive uncertainty

by   Moritz Knolle, et al.

We show that differentially private stochastic gradient descent (DP-SGD) can yield poorly calibrated, overconfident deep learning models. This represents a serious issue for safety-critical applications, e.g. in medical diagnosis. We highlight and exploit parallels between stochastic gradient Langevin dynamics, a scalable Bayesian inference technique for training deep neural networks, and DP-SGD, in order to train differentially private, Bayesian neural networks with minor adjustments to the original (DP-SGD) algorithm. Our approach provides considerably more reliable uncertainty estimates than DP-SGD, as demonstrated empirically by a reduction in expected calibration error (MNIST ∼5-fold, Pediatric Pneumonia Dataset ∼2-fold).


page 1

page 2

page 3

page 4


A Closer Look at the Calibration of Differentially Private Learners

We systematically study the calibration of classifiers trained with diff...

NeuralDP Differentially private neural networks by design

The application of differential privacy to the training of deep neural n...

Differentially private training of residual networks with scale normalisation

We investigate the optimal choice of replacement layer for Batch Normali...

Differentially Private Data Generation Needs Better Features

Training even moderately-sized generative models with differentially-pri...

Differentially Private Coordinate Descent for Composite Empirical Risk Minimization

Machine learning models can leak information about the data used to trai...

Differentially Private Variational Autoencoders with Term-wise Gradient Aggregation

This paper studies how to learn variational autoencoders with a variety ...

DP-MAC: The Differentially Private Method of Auxiliary Coordinates for Deep Learning

Developing a differentially private deep learning algorithm is challengi...