When Relaxations Go Bad: "Differentially-Private" Machine Learning

02/24/2019
∙
by   Bargav Jayaraman, et al.
∙
4
∙

Differential privacy is becoming a standard notion for performing privacy-preserving machine learning over sensitive data. It provides formal guarantees, in terms of the privacy budget, ϵ, on how much information about individual training records is leaked by the model. While the privacy budget is directly correlated to the privacy leakage, the calibration of the privacy budget is not well understood. As a result, many existing works on privacy-preserving machine learning select large values of ϵ in order to get acceptable utility of the model, with little understanding of the concrete impact of such choices on meaningful privacy. Moreover, in scenarios where iterative learning procedures are used which require privacy guarantees for each iteration, relaxed definitions of differential privacy are often used which further tradeoff privacy for better utility. In this paper, we evaluate the impacts of these choices on privacy in experiments with logistic regression and neural network models. We quantify the privacy leakage in terms of advantage of the adversary performing inference attacks and by analyzing the number of members at risk for exposure. Our main findings are that current mechanisms for differential privacy for machine learning rarely offer acceptable utility-privacy tradeoffs: settings that provide limited accuracy loss provide little effective privacy, and settings that provide strong privacy result in useless models.

READ FULL TEXT VIEW PDF

Authors

page 1

page 2

page 3

page 4

Related Research

0
research
∙ 02/14/2020

LinkedIn's Audience Engagements API: A Privacy Preserving Data Analytics System at Scale

We present a privacy system that leverages differential privacy to prote...
Ryan Rogers, et al. ∙
0
research
∙ 08/27/2020

Every Query Counts: Analyzing the Privacy Loss of Exploratory Data Analyses

An exploratory data analysis is an essential step for every data analyst...
Saskia Nuñez von Voigt, et al. ∙
0
research
∙ 08/08/2019

That which we call private

A casual reader of the study by Jayaraman and Evans in USENIX Security 2...
Úlfar Erlingsson, et al. ∙
0
research
∙ 03/04/2021

Quantifying identifiability to choose and audit ε in differentially private deep learning

Differential privacy allows bounding the influence that training data re...
Daniel Bernau, et al. ∙
0
research
∙ 10/20/2019

Leveraging Hierarchical Representations for Preserving Privacy and Utility in Text

Guaranteeing a certain level of user privacy in an arbitrary piece of te...
Oluwaseyi Feyisetan, et al. ∙
37
research
∙ 03/02/2020

Differential Privacy at Risk: Bridging Randomness and Privacy Budget

The calibration of noise for a privacy-preserving mechanism depends on t...
Ashish Dandekar, et al. ∙
0
research
∙ 06/16/2020

Building a Collaborative Phone Blacklisting System with Local Differential Privacy

Spam phone calls have been rapidly growing from nuisance to an increasin...
Daniele Ucci, et al. ∙
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Differential privacy has become a de facto privacy standard for performing analytics over sensitive data. Successful practical implementations are now available for private querying on databases [49] and collecting user statistics from web browsers [19, 67]. Differential privacy has also been adopted by the machine learning community, resulting in many works on privacy-preserving machine learning nearly all of which use some form of differential privacy. These works include designs for differentially-private versions of prominent machine learning algorithms including empirical risk minimization [10, 11] and deep neural networks [61, 1], for both centralized and distributed data settings.

While many methods for achieving differential privacy have been proposed, it is not well understood how to use these methods in practice. In particular, there is little concrete guidance on how to choose an appropriate privacy budget , and limited understanding of how variants of the differential privacy definition designed to improve utility impact privacy in practice. As a result, privacy-preserving machine learning implementations choose relaxed definitions and arbitrary values for as needed to achieve acceptable model utility. For instance, the implementation of Shokri and Shmatikov [61] requires

proportional to the size of the target deep learning model, which could be in the order of few millions. Setting

to such arbitrarily large values severely undermines the privacy guarantees, although there is no consensus on a hard threshold value for above which formal guarantees differential privacy provides become meaningless in practice.

One proposed way to improve utility for a given value is to relax the definition of differential privacy. Several relaxed definitions of differential privacy have been proposed that are shown to provide better utility even for small values [50, 8, 18]. How much privacy leakage these relaxations allow in adversarial scenarios, however, is not well understood. We shed light on this question by evaluating the relaxed differential privacy notions for different choices of values and empirically measuring privacy leakage, including how many individuals are exposed to different attacks.

Contributions. Our main contribution is the evaluation of differential privacy mechanisms for machine learning to understand the impact of different choices of

and different relaxations of differential privacy on both utility and privacy. We focus our evaluation on gradient perturbation mechanisms, which are applicable to a wide class of machine learning algorithms such as empirical risk minimization (ERM) algorithms, which include logistic regression and support vector machines, and deep learning (Section

2.2). Our experiments cover four popular differential privacy relaxations: differential privacy with advanced composition, zero-concentrated differential privacy [8], and Rényi differential privacy [50] (described in Section 2.1). These variations bound the expected privacy loss instead of the worst-case privacy loss, and hence may allow some individuals to be exposed. We evaluate concrete privacy loss using membership inference attacks [62, 76] and attribute inference attacks [76] (Section 3). While the model utility increases with increase in privacy budget , it also increases the success rate of inference attacks. Hence, we aim to find the range of values for which achieves a balance between utility and privacy, and also to evaluate the concrete privacy leakage in terms of individual members exposed. We study both logistic regression and neural network models, on two multi-class classification data sets. Our key findings (Section 4) raise concerns about the practical risks inherent in relaxed differential privacy notions and arbitrary choices of .

Related work. Orthogonal to our work, Ding et al. [13] and Hay et al. [27] evaluate the existing differential privacy implementations for the correctness of implementation. Whereas, we aim to evaluate the choice of privacy budget and the relaxation of differential privacy for the correct implementations. While Carlini et al. [9] also explore the effect of differential privacy against an attacker, they do not explicitly answer what values of should be used nor do they evaluate the privacy leakage of the relaxed definitions. Li et al. [42] raise concerns about relaxing the differential privacy notion in order to achieve better overall utility, but do not evaluate the leakage. We perform thorough evaluation of the differential privacy variations and quantify their leakage for different privacy budgets. The work of Rahman et al. [59] is most related to our work. The authors evaluate the existing differential privacy implementations against membership inference attacks, but do not evaluate the privacy leakage of relaxed variants of differential privacy. Ours is the first work to consider this problem and experimentally show the excess privacy leakage due to the relaxed notions of differential privacy.

2 Differential Privacy for Machine Learning

Next, we review the definition of differential privacy and its relaxed variations. Section 2.2 surveys mechanisms for achieving differentially-private machine learning. Section 2.3 summarizes different applications of differential privacy to machine learning and surveys the choices of for privacy budget and notion of differential privacy used in implementations of differentially-privacy machine learning.

2.1 Background on Differential Privacy

Differential privacy is a probabilistic privacy mechanism that provides an information-theoretic security guarantee. Dwork [16] gives the following definition:

Definition 2.1 (-Differential Privacy).

Given two neighboring data sets and differing by one record, a mechanism preserves -differential privacy if

where is the privacy budget and

is the failure probability.

The quantity is called the privacy loss. When we achieve a strictly stronger notion of -differential privacy. One way to achieve -DP and

-DP is to add noise sampled from Laplace and Gaussian distributions respectively, where the noise is proportional to the

sensitivity of the mechanism :

Definition 2.2 (Sensitivity).

For two neighbouring data sets and differing by one record, sensitivity of is the maximum change in the output of over all possible inputs:

where is a norm of the vector. Throughout this paper we assume -sensitivity which considers the upper bound on the -norm of .

Composition. Differential privacy satisfies a simple composition property: when two mechanisms with privacy budgets and are performed on the same data, together they consume a privacy budget of . Thus, composing multiple differentially-private mechanisms leads to a linear increase in privacy budget , or alternatively, corresponding increases in noise to maintain a fixed total privacy budget.

Advanced Comp. Concentrated (CDP) Zero Concentrated (zCDP) Rényi (RDP)
Expected Loss
Variance of Loss
Convert from -DP - -CDP -zCDP -RDP
Convert to DP - -DP -DP -DP
Composition of -DP Mechanisms
-DP
-DP
-DP -DP
Group privacy for size -
-DP
-DP -DP
Mechanisms Laplace, Gaussian Gaussian Gaussian Gaussian, Laplace
Suitable Settings Composing multiple DP mechanisms When privacy loss is bounded When privacy loss is zero mean For dynamic accounting of privacy loss
Table 1: Comparison of Different Variations of Differential Privacy

. Derived indirectly via zCDP. . Requires sensitivity bound of 1.

Relaxed Definitions. Dwork [17] showed that this linear composition bound on can be reduced at the cost of slightly increasing the failure probability . In essence, this relaxation considers the expected privacy loss of composition of mechanisms instead of the worst-case privacy loss of the individual mechanisms. Dwork defines this as the advanced composition theorem, and proves that it applies to any differentially-private mechanism. Three commonly-used subsequent relaxed versions of differential privacy are Concentrated Differential Privacy [18], Zero Concentrated Differential Privacy [8], and Rényi Differential Privacy [50]. All of these directly bound the expected privacy loss instead of the worst-case privacy loss for composition of multiple mechanisms in order to achieve better utility. However, it is important to consider the actual impact these relaxations have on the privacy leakage, which is a main focus of this paper.

Dwork et al. [18] note that composition of multiple differential private mechanisms result in expected privacy loss which follows a subgaussian distribution. Thus, the expected privacy loss can be directly bounded by controlling the mean and variance of the subgaussian distribution. This reduces the noise that must be added to the individual mechanisms, thereby improving their utility. The authors term this as concentrated differential privacy [18]:

Definition 2.3 (Concentrated Differential Privacy (CDP)).

A randomized algorithm is -concentrated differentially-private if for all pairs of adjacent data sets and ,

where the subgaussian divergence, , is defined such that the expected privacy loss is bounded by

and the standard deviation of the centered subgaussian distribution is bounded by

. Any -DP algorithm satisfies -CDP, however the converse is not true.

A variation on CDP, zero concentrated differential privacy (zCDP) [8] considers that the expected privacy loss is tightly centered around zero mean:

Definition 2.4 (Zero Concentrated Differential Privacy (zCDP)).

A randomized mechanism is -zero-concentrated differentially private if, for all neighbouring data sets and and all ,

where is the -Rényi divergence between the distribution of and the distribution of .

If satisfies -DP, then it also satisfies -zCDP. Further, if provides -zCDP, it is -DP for any . The Rényi divergence allows zCDP to be mapped back to DP, which is not the case for CDP. However, Bun and Steinke [8] give a relationship between CDP and zCDP, which allows an indirect mapping from CDP to DP (Table 1).

Definition 2.5 (Rényi Dfferential Privacy (RDP) [50]).

A randomized mechanism is said to have -Rényi differential privacy of order (which can be abbreviated as -RDP), if for any adjacent data sets , it holds that

The main difference is that CDP and zCDP require a linear bound on all

positive moments of privacy loss, whereas RDP only requires bounding one moment at a time, which allows for a more accurate numerical analysis of privacy loss. If

is an -RDP mechanism, it also satisfies -DP for any .

Table 1 gives a comparison of the relaxed variations of differential privacy. For all the variations, the privacy budget grows sub-linearly with the number of compositions . For group privacy, the privacy budget for CDP and zCDP grows linearly with the group size , whereas it grows exponentially for RDP. As expected, these relaxations leak more information than pure differential privacy, which is evident from the fact that they cannot be mapped to pure -DP for any value of . In other words, there are no worst-case bounds.

Moments Accountant. Motivated by relaxations of differential privacy, Abadi et al. [1] propose the moments accountant

(MA) mechanism for bounding the expected privacy loss of differentially-private algorithms. The moments accountant tries to bound the higher order moments of the privacy loss random variable. Though the authors do not formalize this as a relaxed definition, we note that their moments bound is analogous to the Rényi divergence. In fact, the moments accountant can be considered as an instantiation of Rényi differential privacy. The moments accountant is widely used for differentially-private deep learning due to its availability as a practical framework, which we discuss in Section 

2.3.

2.2 Differential Privacy Methods for ML

In this section, we discuss methods for modifying machine learning algorithms to satisfy differential privacy. We first review the basics of convex optimization problems, such as empirical risk minimization (ERM) algorithms, and show the different methods of achieving differential privacy during the learning process. Next, we briefly discuss the differential privacy methods that are applicable to non-convex optimization problems, including deep learning.

ERM. Given a training data set , where is a feature matrix and is the vector of class labels, an ERM algorithm aims to reduce the convex objective function of the form,

where

is the convex loss function (such as mean square error (MSE) or cross-entropy loss) that measures the training loss for a given

and is a regularization function. Some common examples of regularization functions are penalty, which makes the vector sparse, and penalty, which shrinks the values of vector.

The goal of the algorithm is to find the optimal that minimizes the objective function: . While many first order [14, 78, 37, 58] and second order [43, 40] methods exist to solve this minimization problem, the most basic procedure is gradient descent where we iteratively calculate the gradient of with respect to and update with the gradient information. This process is repeated until or some other termination condition is met.

There are three possible places for adding privacy-preserving noise during this training process, demarcated in Algorithm 1. First, we could add noise to the objective function , which gives us the objective perturbation mechanism (#1 in Algorithm 1). Second, we could add noise to the gradients at each iteration, which gives us the gradient perturbation mechanism (#2). Finally, we can add noise to obtained after the training, which gives us the output perturbation mechanism (#3). While there are other methods of achieving differential privacy such as input perturbation [15], sample-aggregate framework [52], exponential mechanism [48] and teacher ensemble framework [53], we limit our experimental analysis to gradient perturbation since our main objective is to evaluate the relaxed notions of differential privacy which are applicable to gradient perturbation mechanism.

The amount of noise to be added depends on the sensitivity of the machine learning algorithm that determines the noise needed for different DP definitions. For instance, consider logistic regression with regularization penalty. The objective function is of the form:

Assume that the training features are bounded, and . Chaudhuri et al. [11] prove that for this setting, objective perturbation requires sampling noise in the scale of , and output perturbation requires sampling noise in the scale of . The gradient of the objective function is:

which has a sensitivity of . Thus gradient perturbation requires sampling noise in the scale of at each iteration.

Data: Training data set
Result: Model parameters
#1. Add noise here: objective perturbation
for epoch in epochs do
       #2. Add noise here: gradient perturbation
      
      
end for
#3. Add noise here: output perturbation
return
[2ex]
Algorithm 1 Privacy noise mechanisms.

Deep learning. Deep learning follows the same learning procedure as in Algorithm 1, but the objective function is non-convex. As a result, the sensitivity analysis methods of Chaudhuri et al. [11] do not hold for deep learning as they require strong convexity assumption. Hence, their output and objective perturbation methods are not applicable. An alternative approach is to replace the non-convex function with a convex polynomial function [56, 57], and then use the standard objective perturbation. Application of gradient perturbation requires a bound on the gradient norm. Since the gradient norm can be unbounded in deep learning, gradient perturbation can be used after manually clipping the gradients at each iteration. As noted by Abadi et al. [1], norm clipping provides a sensitivity bound on the gradients which is required for generating noise in gradient perturbation.

2.3 Implementing Differential Privacy

Perturbation Data Set
Chaudhuri et al. [11] Output and Objective Adult 45,220 105 0.2
KDDCup99 70,000 119 0.2
Pathak et al. [55] Output Adult 45,220 105 0.2
Hamm et al. [26] Output KDDCup99 493,000 123 1
URL 200,000 50 1
Zhang et al. [80] Objective US 370,000 14 0.8
Brazil 190,000 14 0.8
Jain and Thakurta [34] Objective CoverType 500,000 54 0.5
KDDCup2010 20,000 2M 0.5
Jain and Thakurta [35] Output and Objective URL 100,000 20M 0.1
COD-RNA 60,000 8 0.1
Song et al. [64] Gradient KDDCup99 50,000 9 1
MNIST 60,000 15 1
Wu et al. [72] Output Protein 72,876 74 0.05
CoverType 498,010 54 0.05
Jayaraman et al. [36] Output Adult 45,220 104 0.5
KDDCup99 70,000 122 0.5
Table 2: Simple ERM Methods which achieve High Utility with Low Privacy Budget.

While MNIST is normally a 10-class task, Song et al. [64] use this for ‘1 vs rest’ binary classification.

This section surveys how differential privacy has been used in machine learning applications, with a particular focus on the compromises implementers have made between privacy and utility. While the effective privacy provided by a differential privacy mechanisms depends crucially on the choice of privacy budget , setting the value is still left open to interpretations and higher privacy budgets sacrifice privacy to provide greater utility. Some of the early data analytics works on frequent pattern mining [6, 41]

, decision trees 

[22], private record linkage [31] and recommender systems [47] were able to achieve both high utility and privacy with settings close to 1. These methods rely on finding frequency counts as a sub-routine, and hence provide -differential privacy by either perturbing the counts using Laplace noise or by releasing the top frequency counts using the exponential mechanism [48]. Machine learning, on the other hand, performs much more complex data analysis, and hence requires higher privacy budgets to maintain utility. We categorize the works in this space based on their choice of and the type of DP relaxation they use. First, we cover simple binary classification works that use small privacy budgets (). Then we survey complex classification tasks which seem to require large privacy budgets. Finally, we summarize recent works that aim to perform complex tasks with low privacy budgets by using relaxed privacy notions.

Binary classification. The first practical implementation of a private machine learning algorithm was proposed by Chaudhuri and Monteleoni [10]. They provide a novel sensitivity analysis under strong convexity constraints, allowing them to use output and objective perturbation for binary logistic regression. Chaudhuri et al. [11] subsequently generalized this method for ERM algorithms. This sensitivity analysis method has since been used by many works for binary classification tasks under different learning settings (listed in Table 2). While these applications require small privacy budgets (), they only focus on learning in restricted settings such as learning with low dimensional data, smooth objective functions and strong convexity assumptions, and are only applicable to simple binary classification tasks.

There has also been a considerable progress in generalizing privacy-preserving machine learning to more complex scenarios such as learning in high dimensional settings [34, 35, 65], learning without strong convexity assumptions [66], or relaxing the assumptions on data and objective functions [63, 79, 70]. However, these advances are mainly of theoretical interest and only a few works provide practical instantiations of their proposed method [34, 35].

Task Perturbation Data Set
Jain et al. [33] Online ERM Objective Year 500,000 90 2 10
CoverType 581,012 54 2 10
Iyengar et al. [32] Binary ERM Objective Adult 45,220 104 2 10
Binary ERM KDDCup99 70,000 114 2 10
Multi-Class ERM CoverType 581,012 54 7 10
Multi-Class ERM MNIST 65,000 784 10 10
High Dimensional ERM Gisette 6,000 5,000 2 10
Phan et al. [56, 57] Deep Learning Objective YesiWell 254 30 2 1
MNIST 60,000 784 10 1
Shokri and Shmatikov [61] Deep Learning Gradient MNIST 60,000 1,024 10 369,200
SVHN 100,000 3,072 10 369,200
Zhao et al. [81] Deep Learning Gradient US 500,000 20 2 100
MNIST 60,000 784 10 100
Table 3: Classification Methods for Complex Tasks

Complex learning tasks requiring large privacy budget. All of the above works are limited to convex learning problems with binary classification tasks. Adopting their approaches to more complex learning tasks requires higher privacy budgets (see Table 3). For instance, the online version of ERM as considered by Jain et al. [33] requires as high as 10 to achieve acceptable utility. Which means that theoretically the model’s privacy loss can be as large as . This can allow an adversary to infer the presence or absence of a record from the training set with high confidence. Further, adopting these binary classification methods for multi-class classification tasks requires even higher values. As noted by Wu et al. [72]

, it would require training a separate binary classifier for each class. Finally, high privacy budgets are required for non-convex learning algorithms, such as deep learning 

[61, 81]. Since the output and objective perturbation methods of Chaudhuri et al. [11] are not applicable to non-convex settings, implementations of differentially-private deep learning rely on gradient perturbation in their iterative learning procedure. These methods do not scale to large numbers of training iterations due to the composition theorem of differential privacy which implies that the privacy budget accumulates across iterations. The only exceptions are the works of Phan et al. [56, 57] that replace the non-linear functions in deep learning with polynomial approximations and then apply objective perturbation. With this transformation, they achieve high model utility for , as shown in Table 3. However, we note that this polynomial approximation is a non-standard approach to deep learning which can limit the model’s learning capacity, and thereby affecting the model accuracy for complex tasks.

Machine learning with relaxed DP definitions. To avoid the stringent composition property of differential privacy, several proposed privacy-preserving deep learning methods adopt relaxed privacy definitions based on expected privacy loss rather than worst-case loss (Section 1). Table 4 lists the gradient perturbation based works that use relaxed notions of differential to reduce the overall privacy budget during iterative learning. The utility benefit of using relaxation is evident from the fact that the privacy budget for deep learning algorithms is significantly lesser than the prior works of Shokri and Shmatikov [61] and Zhao et al. [81] which do not use any relaxation. While these relaxed definitions of differential privacy make complex iterative learning feasible for apparently reasonable values, because of the relaxed differential privacy definition, they provide no worst-case guarantees and might lead to more than expected privacy leakage in practice.

The main goal of our study is to evaluate the impact of implementation decisions regarding the privacy budget and relaxed definitions of differential privacy on the concrete privacy leakage that can be exploited by an attacker in practice. We do this by experimenting with various inference attacks, described in the next section.

Perturbation DP Relaxation Data Set
Huang et al. [29] ERM MA Adult 21000 14 2 0.5
Jayaraman et al. [36] ERM zCDP Adult 45,220 104 2 0.5
KDDCup99 70,000 122 2 0.5
Park et al. [54] ERM zCDP and MA Stroke 50,345 100 2 0.5
LifeScience 26,733 10 2 2
Gowalla 1,256,384 2 2 0.01
OlivettiFace 400 4,096 2 0.3
Lee [39] ERM zCDP Adult 48,842 124 2 1.6
US 40,000 58 2 1.6
Brazil 38,000 53 2 1.6
Geumlek et al. [24] ERM RDP Abalone 2,784 9 2 1
Adult 32,561 100 2 0.05
MNIST 7,988 784 2 0.14
Beaulieu et al. [5] Deep Learning MA eICU 4,328 11 2 3.84
TCGA 994 500 2 6.11
Abadi et al. [1] Deep Learning MA MNIST 60,000 784 10 2
CIFAR 60,000 3,072 10 8
Yu et al. [77] Deep Learning MA MNIST 60,000 784 10 21.5
CIFAR 60,000 3,072 10 21.5
Papernot et al. [53] Deep Learning MA MNIST 60,000 784 10 2
SVHN 60,000 3,072 10 8
Geyer et al. [25] Deep Learning MA MNIST 60,000 784 10 8
Bhowmick et al. [7] Deep Learning MA MNIST 60,000 784 10 3
CIFAR 60,000 3,072 10 3
Hynes et al. [30] Deep Learning MA CIFAR 50,000 3,072 10 4
Table 4: Gradient Perturbation based Classification Methods using Relaxed Notion of Differential Privacy

3 Inference Attacks on Machine Learning

This section surveys the two types of inference attacks, membership inference (Section 3.1) and attribute inference (Section 3.2), and explains why they are a suitable metric for evaluating privacy leakage. Section 3.3 briefly summarizes other relevant privacy attacks on machine learning.

3.1 Membership Inference

The aim of a membership inference attack is to infer whether or not a given record is present in the training set. Membership inference attacks can uncover highly sensitive information from training data. An early membership inference attack showed that it is possible to identify individuals contributing DNA to studies that analyze a mixture of DNA from many individuals, using a statistical distance measure to determine if a known individual is in the mixture [28].

Membership inference attacks can either be completely black-box where an attacker only has query access to the target model [62], or can assume that the attacker has full white-box access to the target model, along with some auxillary information [76]. The first membership inference attack on machine learning was proposed by Shokri et al. [62]. They consider an attacker who can query the target model in a black-box way to obtain confidence scores for the queried input. The attacker tries to exploit the confidence score to determine whether the query input was present in the training data. Their attack method involves first training shadow models on a labelled data set, which can be generated either via black-box queries to the target model or through assumptions about the underlying distribution of training set. The attacker then trains an attack model using the shadow models to distinguish whether or not an input record is in the shadow training set. Finally, the attacker makes API calls to the target model to obtain confidence scores for each given input record and infers whether or not the input was part of the target model’s training set. The inference model distinguishes between the target model’s predictions for inputs that are in its training set and those it did not train on. The key assumption is that the confidence score of the target model is higher for the training instances than it would be for arbitrary instances not present in the training set. This can be due to the generalization gap, which is prominent in models that overfit to training data.

A more targeted approach was proposed by Long et al. [44] where the shadow models are trained with and without a targeted input record . At inference time, the attacker can check if the input record was present in the training set of target model. This approach tests the membership of a specific record more accurately than Shokri et al.’s approach [62]. Recently, Salem et al. [60] proposed more generic membership inference attacks by relaxing the requirements of Shokri et al. [62]. In particular, requirements on the number of shadow models, knowledge of training data distribution and the target model architecture can be relaxed without substantially degrading the effectiveness of the attack.

Yeom et al. [76] recently proposed a more computationally efficient membership inference attack when the attacker has access to the target model and knows the average training loss of the model. To test the membership of an input record, the attacker evaluates the loss of the model on the input record and then classifies it as a member if the loss is smaller than the average training loss.

Connection to Differential Privacy. Differential privacy, by definition, aims to obfuscate the presence or absence of a record in the data set. On the other hand, membership inference attacks aim to identify the presence or absence of a record in the data set. Thus, intuitively these two notions are competing each other. Li et al. [42] point to this fact and provide a direct relationship between differential privacy and membership inference attacks. Moreover, Backes et al. [3] studied membership inference attacks on microRNA studies and showed that differential privacy can reduce the success of membership inference attacks, but at the cost of utility.

Yeom et al. [76] formally define a membership inference attack as an adversarial game where a data element is selected from the distribution, which is randomly either included in the training set or not. Then, an adversary with access to the trained model attempts to determine if that element was used in training. The membership advantage is defined as the difference between the adversary’s true and false positive rates for this game. The authors prove that if the learning algorithm satisfies -differential privacy, then the adversary’s advantage is bounded by . Hence, it is natural to use membership inference attacks as a metric to evaluate the privacy leakage of differentially-private algorithms.

3.2 Attribute Inference

The aim of an attribute inference attack (also called model inversion) is to learn hidden sensitive attributes of a test input given at least API access to the model and information about the non-sensitive attributes. Fredrikson et al. [21]

formalize this attack in terms of maximizing the posterior probability estimate of the sensitive attribute. More concretely, for a test record

where the attacker knows the values of its non-sensitive attributes

and all the prior probabilities of the attributes, the attacker obtains the output of the model,

, and attempts to recover the value of the sensitive attribute . The attacker essentially searches for the value of that maximizes the posterior probability . The success of this attack is based on the correlation between the sensitive attribute, , and the model output, .

Yeom et al. [76] also propose an attribute inference attack using the same principle they use for their membership inference attack. The attacker evaluates the model’s empirical loss on the input instance for different values of the sensitive attribute, and reports the value which has the maximum posterior probability of achieving the empirical loss. The authors define the attribute advantage similarly to their definition of membership advantage for membership inference.

Fredrikson et al. [21] demonstrated attribute inference attacks that could identifying genetic markers based on warfarin dosage output by a model with just black-box access to model API.111This application has stirred some controversy based on the warfarin dosage output by the model itself being sensitive information correlated to the sensitive genetic markers, hence the assumption on attacker’s prior knowledge of warfarin dosage is somewhat unrealistic [46]. With additional access to confidence scores of the model (noted as white-box information by Wu et al. [71]), more complex tasks have been performed, such as recovering faces from the training data [20].

Connection to Differential Privacy. Differential privacy is mainly tailored to obfuscate the presence or absence of a record in a data set, by limiting the effect of any single record on the output of differential private model trained on the data set. Logically this definition also extends to attributes or features of a record. In other words, by adding sufficient differential privacy noise, we should be able to limit the effect of a sensitive attribute on the model’s output. This relationship between records and attributes is discussed by Yeom et al. [76]. Hence, we include these attacks in our experiments.

3.3 Other Attacks on Machine Learning

Apart from inference attacks, many other attacks have been proposed in the literature which try to infer specific information from the target model. These attacks include model stealing, hyperparameter stealing, property inference and memorization. Among these, memorization is the closest to membership inference attack. It tries to exploit the ability of high capacity models, such as deep learning models, to memorize certain sensitive patterns in the training data [9]. However, these attacks are shown to be easily thwarted by differential privacy mechanisms even with very small noise ().

Model stealing attacks aim to recover the model parameters via black-box access to the target model, either by adversarial learning [45] or by equation solving attacks [68]. Hyperparameter stealing attacks try to recover the underlying hyperparameters used during the model training, such as regularization coefficient [69] or model architecture [74]. These hyperparameters are intellectual property of commercial organizations that deploy machine learning models as a service, and hence these attacks are regarded as a threat to valuable intellectual property. A property inference attack tries to infer whether the training data set has a specific property, given a white-box access to the trained model. For instance, given access to a speech recognition model, an attacker can infer if the training data set contains speakers with a certain accent. Here the attacker can use the shadow training method of Shokri et al. [62] for distinguishing the presence and absence of a target property. These attacks have been performed on HMM and SVM models [2] and neural networks [23].

Though all these attacks may leak sensitive information about the target model or training data, the information the leak tends to be application-specific and is not clearly defined in a general way. For example, a property inference attack leaks some statistical property of the training data that is surprising to the model developer; but, the overall purpose of the model is to learn statistical properties from the training data, so there is no general definition of a property inference attack without a prescriptive decision about which statistical properties of the training data should be captured by the model and which are sensitive to leak. In addition, the attacks mentioned in this section do not closely follow the threat model of differential privacy. Thus, we only consider inference attacks for our experimental evaluation. In addition to these attacks, several poisoning and adversarial training attacks have been proposed [73, 51, 4, 75] which require an adversary that can actively interfere with the model training process. We consider these out of scope for this paper, and assume a clean training process not under the control of the adversary.

4 Empirical Evaluation

To quantify the privacy leakage of the differentially-private implementations for machine learning, we conduct experiments to measure how much an adversary can infer from a model. While differential privacy can mitigate most privacy attacks (except for property inference attacks), as motivated in Section 3, we measure privacy leakage using membership and attribute inference in our experiments. Note, however, that the conclusions we can draw from experiments like this are limited to showing a lower bound on the information leakage since they are measuring the effectiveness of a particular implementation of an attack. Experimental results from particular attacks cannot be used to make strong claims about what the best possible attack would be able to infer, especially in cases where an adversary has auxiliary information to help guide the attack. Results from our experiments, however, do provide clear evidence for when privacy protections do not appear to improve privacy in a meaningful way, and, in particular, where the expected privacy claims of relaxed mechanisms do not hold up in practice.

4.1 Experimental Setup

We evaluate the privacy leakage of two differentially-private algorithms using gradient perturbation: logistic regression for empirical risk minimization (Section 4.2) and neural networks for non-convex learning (Section 4.3). For both, we consider the different relaxed notions of differential privacy and compare their privacy leakage. The variations that we implement are naïve composition, advanced composition, zero concentrated differential privacy (zCDP) and Rényi differential privacy (RDP) (see Section 1 for details). We do not include CDP as it has the same composition property as zCDP (Table 1). For RDP, we use , using -Rényi divergence to bound the privacy loss.

We evaluate each most on two main metrics: accuracy loss, where we measure the model’s accuracy loss on test set with respect to the non-private baseline, and privacy leakage, where we measure the attacker’s advantage as defined by Yeom et al. [76]. This corresponds to the difference between the true positive rate (TPR) and the false positive rate (FPR) of the inference attack. To better understand the potential impact of leakage, we also conduct experiments to measure the actual number of members who are at risk for disclosure in a membership inference attack.

Data sets. We evaluate our models over two datasets for multi-class classification tasks: CIFAR-100 [38] and Purchase-100 [12]. CIFAR-100 consists of images of 100 real world objects, with 500 instances of each object class. We use PCA to reduce the dimensionality of records to 50. The Purchase-100 data set consists of 200,000 customer purchase records of size 100 each (corresponding to the 100 most frequently-purchased items) where the records are grouped into 100 classes based on the customers’ purchase style. For both data sets, we use 10,000 randomly-selected instances for training and 10,000 randomly-selected non-training instances for the test set. The remaining records are used for training shadow models and inference model.

Attacks. For our experiments, we use the attack frameworks of Shokri et al. [62] and Yeom et al. [76] for membership inference and the method proposed by Yeom et al. [76] for attribute inference. In Shokri et al.’s framework [62], multiple shadow models are trained on data that is sampled from the same distribution as the private data set. These shadow models are used to train an inference model to identify whether an input record belongs to the private data set. The inference model is trained using a set of records used to train the shadow models, a set of records randomly selected from the distribution that are not part of the shadow model training, along with the confidence scores output by the shadow models for all of the input records. Using these inputs, the inference model learns to distinguish the training records from the non-training records. At the inference stage, the inference model takes an input record along with the confidence score of the target model on the input record, and outputs whether the input record belongs to the target model’s private training data set. The intuition is that if the target model overfits on its training set, its confidence score for a training record will be higher than its confidence score for an otherwise similar input that was not used in training. The inference model tries to exploit this property. In our instantiation of the attack framework, we use five shadow models which all have the same model architecture as the target model. Our inference model is a neural network with two hidden layers of size 64. This setting is in accordance to the original work [62].

The attack framework of Yeom et al. [76] is simpler than Shokri et al.’s design. It assumes a white-box attacker with access to the target model’s expected training loss on the private data set, in addition to having access to the target model. For membership inference, the attacker simply observes the target model’s loss on the input record. The attacker classifies the record as a member of private data set if the loss is smaller than the target model’s expected training loss, otherwise the record is classified as a non-member. The same principle is used for attribute inference. Given an input record, the attacker brute-forces all possible values for the unknown private attribute and observes the target model’s loss, outputting the value for which the loss is closest to the target’s expected training loss. Since there are no attributes in our data sets that are explicitly annotated as private, we randomly choose five attributes, and perform the attribute inference attack on each attribute independently, and report the averaged results.

4.2 Logistic Regression Results

For both the CIFAR-100 and Purchase-100 data sets, we train a regularized logistic regression model with regularization for strong convexity. First, we train a non-private model and perform a grid search over the regularization coefficient to find the value that minimizes the classification error on test set. Next, we fix this setting to train differentially-private models using gradient perturbation. We vary between 0.5 to 1000, and report the accuracy loss and privacy leakage

. Due to the random noise addition, all the experiments are repeated five times and the average results and standard errors are reported.

We note that our gradient perturbation implementations require higher privacy budgets for achieving model accuracy compared to what was reported by Abadi et al. [1] and Yu et al. [77]. While their implementations achieve model accuracy close to non-private baselines for , our models require . The reason for this is because we do not assume pre-trained model parameters, unlike the prior works. We performed some initial experiments on Purchase-100 with the setting of Abadi et al. [1]

where we pre-train our model on a separate hold-out data set, and then privately train on the sensitive data set with gradient clipping threshold of

(Abadi et al. use ). While smaller value of corresponds to smaller noise added, it also limits the information in the gradients leading to poor or no training. Without the pre-training, we observed that the private training at achieves accuracy of only 0.033 on test set. Whereas with pre-training, the model achieves test accuracy of 0.736 after private training. The model achieves test accuracy of 0.738 by just training on the hold-out set. So, it is evident that the model is not learning anything from the private data set, and the model performance is solely due to the pre-training on the hold-out set. Hence, we require the usage of large values of to perform any useful learning on private data. For our experiments, we scale with to avoid separate hyperparameter tuning. However, in practice it is recommended to either fix based on the average gradient norm for the data set, or to do extensive hyperparameter tuning. Both the approaches would require allocating part of the privacy budget for hyperparameter tuning to ensure differential privacy.

CIFAR-100 results. The baseline non-private logistic regression model achieves accuracy of 0.207 on training set and 0.150 on test set, which is competitive to the state-of-art neural network model [62] that achieves test accuracy close to 0.20 on CIFAR-100 after training on larger data set. Thus there is a small generalization gap of 0.057, which the inference attacks try to exploit. Figure 1 compares the accuracy loss for logistic regression models trained with different relaxed notions of differential privacy as we varying the privacy budget . The accuracy loss is normalized with respect to the accuracy of non-private model to clearly depict the model utility. An accuracy loss value of 1 means that the model has 100% loss and hence has no utility, whereas the value of 0 means that the model achieves same accuracy as the non-private baseline. As depicted in the figure, none of the variants learn anything useful for . For , RDP and zCDP achieve lower accuracy loss than the other methods, as expected since the relaxed definitions allow for less added noise. Notice that for , all the models achieve accuracy close to 0.01 which is random guessing for 100 class classification. For this example, there is no choice of available that provides any effective privacy for a model that does better than random guessing.

Figure 1: Accuracy loss of logistic regression (CIFAR-100).
Figure 2: Shokri et al. membership inference (logistic regression, CIFAR-100).
Figure 3: Yeom et al. membership inference (logistic regression, CIFAR-100).

Figures 2 and 3 show the privacy leakage due to membership inference attacks on logistic regression models. We provide the attacker with a set of records half of which are from the training set. We call records in the training set members, and the remaining records non-members. The task of the attacker is to predict whether or not a given input record belongs to the training set (i.e., if it is a member). Figure 2 shows results for the black-box attacker of Shokri et al. [62], which has access to the target model’s confidence scores on the input record. For all of the variations of differential privacy, the privacy leakage for is minimal and similar (attacker advantage is close to 0.01 for ). For , naïve composition has leakage of , whereas the relaxed variants have average leakage close to .

Naive Composition Advanced Composition zCDP RDP
Loss 1% 2% 5% Loss 1% 2% 5% Loss 1% 2% 5% Loss 1% 2% 5%
0.1 0.94 0 0 0 0.93 0 0 0 0.94 0 0 0 0.94 0 0 0
0.5 0.93 0 0 0 0.94 0 0 0 0.93 0 0 0 0.93 0 0 1
1.0 0.94 0 0 0 0.93 0 0 0 0.92 0 0 0 0.93 0 0 0
5.0 0.94 0 0 0 0.92 0 0 0 0.91 0 0 1 0.92 0 0 0
10.0 0.93 0 0 0 0.92 0 0 0 0.90 0 0 0 0.89 0 0 0
50.0 0.92 0 0 0 0.81 0 0 1 0.65 6 12 41 0.66 4 7 35
100.0 0.89 0 0 0 0.62 1 3 25 0.43 28 67 175 0.47 19 48 138
500.0 0.30 23 45 169 0.07 103 243 606 0.06 109 263 653 0.06 101 249 620
1000.0 0.11 54 103 346 0.04 106 251 616 0.04 115 279 676 0.04 105 259 616
Table 5: Number of individuals (out of 10,000) exposed by Yeom et al. membership inference attack on logistic regression (CIFAR-100). The non-private () model leaks 145, 265 and 704 members for 1%, 2% and 5% FPR respectively.

Figure 3 shows results for the white-box attacker of Yeom et al. [62], which has access to the target model’s loss on the input record. The difference between the relaxed variants is more apparent for this attack compare to the membership inference attack in Figure 2. This is because the attacker in Shokri et al. [62] is restricted by the attack model’s learning capacity. As expected, zCDP and RDP relaxations leak the most, followed by advanced composition. Naïve composition does not have any leakage for , but the leakage reaches for . The observed leakage of all the variations is in accordance with the noise magnitude required for different differential privacy guarantees.

Figure 4 depicts the privacy leakage due to the attribute inference attack. As shown, the privacy leakage of zCDP is highest, closely followed by RDP. The relaxed variants have privacy leakage comparable to naïve composition for , and tend to leak more for larger values of . Naïve composition has low privacy leakage for (attacker advantage of at ), but it quickly increases to for . But for meaningful privacy budgets there is no significant leakage () for any of the methods. From Figures 1–4, we see as expected that as privacy budgets increase, the attacker’s advantage (privacy leakage) increases and the model utility increases (accuracy loss decreases). This trend is consistent across all variations.

Figure 4: Attribute inference (logistic regression, CIFAR-100).

To gain more understanding of the impact of privacy leakage, we investigate the actual number of training set members exposed to the attacker for different differential privacy variations. We assume the attacker has some limited tolerance for falsely exposing a member (that is, a bound on the acceptable false positive rate), and sets the required threshold score for the inference model output as the level needed to achieve that false positive rate. Then, we count the number of members in the private training data set for whom the inference model output exceeds that confidence threshold. Table 5 summarizes the results, reporting the number of members exposed to an adversary who tolerates false positive rates of 1%, 2%, and 5%. As we increase the tolerance threshold, there is a gradual increase in membership leakage for all the methods. While all the methods are resistant to attack for , the leakage of relaxed variants increases drastically for higher values.

Purchase-100 results. The baseline non-private logistic regression model achieves accuracy of 0.8127 on the training set and 0.6322 on test set. In comparison, Google ML platform’s black-box trained model achieves a test accuracy of 0.656 for Purchase-100 (see Shokri et al. [62] for details). Since the results are similar to the CIFAR-100 data set, we only show the plots comparing the accuracy loss and privacy leakage of models against the membership inference attack of Yeom et al. [76] for this data set.

Figure 5 shows the accuracy loss of all variants on Purchase-100 data set. None of the variants have any utility for . For , naïve composition achieves accuracy loss of , while all the other variants achieve accuracy loss close to 0.2 (zCDP achieves the least loss of ). None of the variants achieve model utility close to the non-private baseline for any privacy budget.

Figure 5: Accuracy loss of logistic regression (Purchase-100).

Figure 6 shows the privacy leakage comparison of the variants against membership inference attack on Purchase-100 data set. The leakage is in accordance to the noise each variant adds and it increases proportional to the model utility. Hence, if a model has reasonable utility, it is bound to leak membership information. Table 7 (in the Appendix) shows the number of individual members exposed, with similar results to the findings for CIFAR-100.

Figure 6: Membership inference attack [76] on logistic regression (Purchase-100).

4.3 Neural Networks

We train a neural network model consisting of two hidden layers and an output layer. The hidden layers have 256 neurons that use

ReLU activation. The output layer is a softmax layer with 100 neurons, each corresponding to a class label. This architecture is similar to the one used by Shokri et al. [62].

CIFAR-100 results. The baseline non-private neural network model achieves accuracy of 0.9856 on the training set and 0.1788 on test set, which is competitive to the neural network model of Shokri et al. [62]. Their model is trained on training set of size 29,540 and achieves test accuracy of 0.20, whereas our model is trained on 10,000 training instances. Thus, there is a huge generalization gap of 0.8068, which the inference attacks can exploit. Figure 7 shows the accuracy loss comparison of neural network models trained with different relaxed notions of differential privacy with varying privacy budget . While the relaxed variants begin gaining accuracy for , the model trained with naïve composition does not learning anything useful until , at which point the other variants achieve accuracy loss close to zero.

Figure 7: Accuracy loss for neural network (CIFAR-100).

Figure 8 shows the privacy leakage due to membership inference attacks on neural network models trained with different relaxed notions for both attacks. The privacy leakage for all the variations of differential privacy is close to zero for . For , with the Shokri et al. attack, naïve composition has leakage of compared to for advanced composition, for zCDP, and for RDP. For the white-box attacker of Yeom et al. [76], the zCDP and RDP relaxations begin leaking privacy at whereas naïve composition does not leak for any value of . RDP leaks the most for (membership advantage of ). This is because these relaxed variations consider only expected privacy loss, and hence add considerably less noise in comparison to naïve composition. Naïve composition achieves strong privacy against membership inference attackers, but fails to learning anything useful. No option appears to provide both reasonable model utility and meaningful privacy.

Naive Composition Advanced Composition zCDP RDP
Loss 1% 2% 5% Loss 1% 2% 5% Loss 1% 2% 5% Loss 1% 2% 5%
0.1 0.95 0 0 0 0.95 0 0 0 0.94 0 0 0 0.93 0 0 0
0.5 0.94 0 0 0 0.94 0 0 0 0.93 0 0 0 0.93 0 5 90
1.0 0.94 0 0 0 0.94 0 0 0 0.92 0 0 6 0.91 0 4 94
5.0 0.94 0 0 0 0.93 0 0 0 0.83 0 3 16 0.83 0 0 45
10.0 0.94 0 0 0 0.87 0 0 1 0.81 0 0 20 0.80 0 11 109
50.0 0.95 0 0 0 0.73 0 0 12 0.64 0 1 87 0.64 0 1 70
100.0 0.93 0 0 0 0.61 1 8 32 0.49 30 75 281 0.48 11 43 202
500.0 0.93 0 0 2 0.06 26 76 331 0.00 54 119 399 0.00 40 104 379
1000.0 0.59 0 0 11 0.06 13 53 359 0.00 28 78 416 0.07 22 64 383
Table 6: Number of members (out of 10,000) exposed by Yeom et al. membership inference attack on neural network (CIFAR-100).

Non-private model leaks 155, 425 and 2667 members for 1%, 2% and 5% FPR respectively.

Figure 8: Yeom et al. membership inference attack (neural network, CIFAR-100).
Figure 9: Attribute inference (neural network, CIFAR-100).

Figure 9 depicts the privacy leakage due to attribute inference attack on neural network models trained with different relaxed notions. Again zCDP and RDP are more vulnerable to attribute inference attack due to their relaxation of differential privacy guarantee. Naïve composition is resistant to the attack for . In comparison to the case of logistic regression, here the relaxed variants have more privacy leakage as the generalization gap for neural network is large.

As for logistic regression, we further investigate the actual number of training set members exposed to the attacker and report the results in Table 6. The impact ot the increased leakage of relaxed definitions is much more severe for neural networks, revealing members at the 2% false positive threshold for RDP even at . Naïve composition is secure even with high privacy budgets, revealing no members at 2% false positive threshold for any value, even though the non-private model leaks between 155 members at 1% false positive threshold. On the other hand, the relaxed variants of differential privacy expose many members, even with fairly low values, and expose hundreds of members at the privacy budgets needed to achieve low accuracy loss. As shown, zCDP leaks more members than RDP for . This is because RDP adds more noise when . Since we fix , RDP perturbs more than zCDP for .

Figure 10: Accuracy loss of neural network (Purchase-100).

Purchase-100 results. The baseline non-private neural network model achieves accuracy of 0.9998 on the training set and 0.7438 on test set. Our model performs better than the neural network model Shokri et al. [62] trained on the same data set which reports 0.670 test accuracy. Since most of the results are similar to the results on CIFAR-100 data set, we exclude the plots on privacy leakage comparison of differential privacy variants against the attribute inference attack of Yeom et al. [76] and the membership inference attack of Shokri et al. [62]. We only show the accuracy loss (Figure 10) and privacy leakage results for membership inference attack of Yeom et al. [76]. The trends for both accuracy and privacy are similar to that of logistic regression model on Purchase-100 data set (Figure 5). However, here the relaxed variants achieve model utility close to the non-private baseline for , while naïve composition achieves accuracy loss of . Figure 11 (in the Appendix) shows the privacy leakage comparison of the variants against membership inference attack and Table 8 shows the number of individuals exposed. The results are consistent with those observed for CIFAR-100.

5 Conclusion

Differential privacy has earned a well-deserved reputation providing a principled and powerful mechanisms for ensuring provable privacy. However, when it is implemented for challenging tasks such as machine learning, compromises must be made to preserve utility. It is essential that the privacy impact of those compromises is well understood by implementers deploying differential privacy on sensitive data. Our results are a step towards improving that understanding, and reveal that the commonly used relaxations of differential privacy may provide unacceptable utility-privacy tradeoffs. We hope our study will encourage more careful assessments of the practical privacy value of formal claims based on differential privacy, and lead to work on deeper understanding of the privacy impact of design decisions when deploying differential privacy, and eventually to solutions that provide desirable, and well understood, utility-privacy tradeoffs.

Availability

Open source code for reproducing all of our experiments is available at https://github.com/bargavj/EvaluatingDPML.

Acknowledgments

This work was partially funded by a grant from the National Science Foundation (#1717950). We would like to thank Atallah Hezbor, Faysal Hossain Shezan, Tanmoy Sen, Max Naylor, Joshua Holtzman and Nan Yang for helping in systematizing the related works. Finally, we would also like to thank Congzheng Song and Samuel Yeom for providing their implementation of inference attacks.

References

Appendix A Additional results for Purchase-100

Figure 11: Yeom et al. membership inference attack (neural network, Purchase-100).
Naïve Composition Advanced Composition zCDP RDP
Loss 1% 2% 5% Loss 1% 2% 5% Loss 1% 2% 5% Loss 1% 2% 5%
0.1 0.99 0 0 0 0.98 0 0 0 0.98 0 0 0 0.98 0 0 0
0.5 0.98 0 0 0 0.99 0 0 0 0.98 0 0 0 0.98 0 0 0
1.0 0.99 0 0 0 0.98 0 0 0 0.98 0 0 0 0.97 0 0 0
5.0 0.98 0 0 0 0.97 0 0 8 0.98 0 1 85 0.97 1 5 101
10.0 0.98 0 0 0 0.97 0 2 20 0.97 0 2 101 0.97 1 3 105
50.0 0.97 0 0 0 0.96 0 1 39 0.95 0 9 205 0.95 4 12 138
100.0 0.97 0 0 0 0.95 0 9 90 0.92 5 30 210 0.93 3 20 163
500.0 0.87 0 0 2 0.60 37 76 216 0.55 64 131 390 0.58 49 84 268
1000.0 0.63 3 7 22 0.22 89 182 435 0.19 118 231 582 0.23 87 156 411
0.00 212 363 957 0.00 212 363 957 0.00 212 363 957 0.00 212 363 957
Table 7: Number of members (out of 10,000) exposed by Yeom et al. membership inference attack on logistic regression (Purchase-100).
Naïve Composition Advanced Composition zCDP RDP
Loss 1% 2% 5% Loss 1% 2% 5% Loss 1% 2% 5% Loss 1% 2% 5%
0.1 0.99 0 0 0 0.99 0 0 0 0.99 0 0 0 0.98 0 0 0
0.5 0.99 0 0 0 0.98 0 0 0 0.98 0 0 0 0.98 0 0 1
1.0 0.99 0 0 0 0.98 0 0 1 0.98 0 0 0 0.98 0 0 0
5.0 0.98 0 0 0 0.98 0 0 0 0.96 0 11 163 0.96 0 13 233
10.0 0.99 0 0 0 0.96 0 0 0 0.96 0 4 104 0.96 1 24 219
50.0 0.98 0 0 0 0.91 0 0 1 0.94 11 37 147 0.94 3 20 168
100.0 0.98 0 0 0 0.86 0 0 0 0.85 21 59 221 0.85 9 42 176
500.0 0.95 36 74 107 0.06 31 61 166 0.12 50 98 256 0.06 49 74 196
1000.0 0.65 0 0 0 0.02 0 0 217 0.02 54 102 266 0.02 0 0 198
0.00 0 196 574 0.00 0 196 574 0.00 0 196 574 0.00 0 196 574
Table 8: Number of members (out of 10,000) exposed by membership inference attack [76] on neural network (Purchase-100).

First column for each method depicts the accuracy loss with respect to non-private model.

Sign up for DeepAI

Join one of the world's largest A.I. communities

Already have an account? Login here

Login to DeepAI

Don't have an account? Signup here

SHARE