DeepAI
Log In Sign Up

Improving Deep Learning with Differential Privacy using Gradient Encoding and Denoising

07/22/2020
by   Milad Nasr, et al.
0

Deep learning models leak significant amounts of information about their training datasets. Previous work has investigated training models with differential privacy (DP) guarantees through adding DP noise to the gradients. However, such solutions (specifically, DPSGD), result in large degradations in the accuracy of the trained models. In this paper, we aim at training deep learning models with DP guarantees while preserving model accuracy much better than previous works. Our key technique is to encode gradients to map them to a smaller vector space, therefore enabling us to obtain DP guarantees for different noise distributions. This allows us to investigate and choose noise distributions that best preserve model accuracy for a target privacy budget. We also take advantage of the post-processing property of differential privacy by introducing the idea of denoising, which further improves the utility of the trained models without degrading their DP guarantees. We show that our mechanism outperforms the state-of-the-art DPSGD; for instance, for the same model accuracy of 96.1% on MNIST, our technique results in a privacy bound of ϵ=3.2 compared to ϵ=6 of DPSGD, which is a significant improvement.

READ FULL TEXT

page 1

page 2

page 3

page 4

06/17/2021

Accuracy, Interpretability, and Differential Privacy via Explainable Boosting

We show that adding differential privacy to Explainable Boosting Machine...
12/15/2018

A General Approach to Adding Differential Privacy to Iterative Training Procedures

In this work we address the practical challenges of training machine lea...
05/28/2019

Differential Privacy Has Disparate Impact on Model Accuracy

Differential privacy (DP) is a popular mechanism for training machine le...
10/16/2021

DPNAS: Neural Architecture Search for Deep Learning with Differential Privacy

Training deep neural networks (DNNs) for meaningful differential privacy...
08/14/2020

Three Variants of Differential Privacy: Lossless Conversion and Applications

We consider three different variants of differential privacy (DP), namel...
02/13/2019

Privacy-Utility Trade-off of Linear Regression under Random Projections and Additive Noise

Data privacy is an important concern in machine learning, and is fundame...
11/29/2018

The Power of The Hybrid Model for Mean Estimation

In this work we explore the power of the hybrid model of differential pr...