1 Introduction
Differential privacy has become a de facto privacy standard for performing analytics over sensitive data. Successful practical implementations are now available for private querying on databases [49] and collecting user statistics from web browsers [19, 67]. Differential privacy has also been adopted by the machine learning community, resulting in many works on privacy-preserving machine learning nearly all of which use some form of differential privacy. These works include designs for differentially-private versions of prominent machine learning algorithms including empirical risk minimization [10, 11] and deep neural networks [61, 1], for both centralized and distributed data settings.
While many methods for achieving differential privacy have been proposed, it is not well understood how to use these methods in practice. In particular, there is little concrete guidance on how to choose an appropriate privacy budget , and limited understanding of how variants of the differential privacy definition designed to improve utility impact privacy in practice. As a result, privacy-preserving machine learning implementations choose relaxed definitions and arbitrary values for as needed to achieve acceptable model utility. For instance, the implementation of Shokri and Shmatikov [61] requires
proportional to the size of the target deep learning model, which could be in the order of few millions. Setting
to such arbitrarily large values severely undermines the privacy guarantees, although there is no consensus on a hard threshold value for above which formal guarantees differential privacy provides become meaningless in practice.One proposed way to improve utility for a given value is to relax the definition of differential privacy. Several relaxed definitions of differential privacy have been proposed that are shown to provide better utility even for small values [50, 8, 18]. How much privacy leakage these relaxations allow in adversarial scenarios, however, is not well understood. We shed light on this question by evaluating the relaxed differential privacy notions for different choices of values and empirically measuring privacy leakage, including how many individuals are exposed to different attacks.
Contributions. Our main contribution is the evaluation of differential privacy mechanisms for machine learning to understand the impact of different choices of
and different relaxations of differential privacy on both utility and privacy. We focus our evaluation on gradient perturbation mechanisms, which are applicable to a wide class of machine learning algorithms such as empirical risk minimization (ERM) algorithms, which include logistic regression and support vector machines, and deep learning (Section
2.2). Our experiments cover four popular differential privacy relaxations: differential privacy with advanced composition, zero-concentrated differential privacy [8], and Rényi differential privacy [50] (described in Section 2.1). These variations bound the expected privacy loss instead of the worst-case privacy loss, and hence may allow some individuals to be exposed. We evaluate concrete privacy loss using membership inference attacks [62, 76] and attribute inference attacks [76] (Section 3). While the model utility increases with increase in privacy budget , it also increases the success rate of inference attacks. Hence, we aim to find the range of values for which achieves a balance between utility and privacy, and also to evaluate the concrete privacy leakage in terms of individual members exposed. We study both logistic regression and neural network models, on two multi-class classification data sets. Our key findings (Section 4) raise concerns about the practical risks inherent in relaxed differential privacy notions and arbitrary choices of .Related work. Orthogonal to our work, Ding et al. [13] and Hay et al. [27] evaluate the existing differential privacy implementations for the correctness of implementation. Whereas, we aim to evaluate the choice of privacy budget and the relaxation of differential privacy for the correct implementations. While Carlini et al. [9] also explore the effect of differential privacy against an attacker, they do not explicitly answer what values of should be used nor do they evaluate the privacy leakage of the relaxed definitions. Li et al. [42] raise concerns about relaxing the differential privacy notion in order to achieve better overall utility, but do not evaluate the leakage. We perform thorough evaluation of the differential privacy variations and quantify their leakage for different privacy budgets. The work of Rahman et al. [59] is most related to our work. The authors evaluate the existing differential privacy implementations against membership inference attacks, but do not evaluate the privacy leakage of relaxed variants of differential privacy. Ours is the first work to consider this problem and experimentally show the excess privacy leakage due to the relaxed notions of differential privacy.
2 Differential Privacy for Machine Learning
Next, we review the definition of differential privacy and its relaxed variations. Section 2.2 surveys mechanisms for achieving differentially-private machine learning. Section 2.3 summarizes different applications of differential privacy to machine learning and surveys the choices of for privacy budget and notion of differential privacy used in implementations of differentially-privacy machine learning.
2.1 Background on Differential Privacy
Differential privacy is a probabilistic privacy mechanism that provides an information-theoretic security guarantee. Dwork [16] gives the following definition:
Definition 2.1 (-Differential Privacy).
Given two neighboring data sets and differing by one record, a mechanism preserves -differential privacy if
where is the privacy budget and
is the failure probability.
The quantity is called the privacy loss. When we achieve a strictly stronger notion of -differential privacy. One way to achieve -DP and
-DP is to add noise sampled from Laplace and Gaussian distributions respectively, where the noise is proportional to the
sensitivity of the mechanism :Definition 2.2 (Sensitivity).
For two neighbouring data sets and differing by one record, sensitivity of is the maximum change in the output of over all possible inputs:
where is a norm of the vector. Throughout this paper we assume -sensitivity which considers the upper bound on the -norm of .
Composition. Differential privacy satisfies a simple composition property: when two mechanisms with privacy budgets and are performed on the same data, together they consume a privacy budget of . Thus, composing multiple differentially-private mechanisms leads to a linear increase in privacy budget , or alternatively, corresponding increases in noise to maintain a fixed total privacy budget.
Advanced Comp. | Concentrated (CDP) | Zero Concentrated (zCDP) | Rényi (RDP) | |
---|---|---|---|---|
Expected Loss | ||||
Variance of Loss | ||||
Convert from -DP | - | -CDP | -zCDP | -RDP |
Convert to DP | - | -DP | -DP | -DP |
Composition of -DP Mechanisms | ||||
-DP | ||||
-DP | ||||
-DP | -DP | |||
Group privacy for size | - | |||
-DP | ||||
-DP | -DP | |||
Mechanisms | Laplace, Gaussian | Gaussian | Gaussian | Gaussian, Laplace |
Suitable Settings | Composing multiple DP mechanisms | When privacy loss is bounded | When privacy loss is zero mean | For dynamic accounting of privacy loss |
. Derived indirectly via zCDP. . Requires sensitivity bound of 1.
Relaxed Definitions. Dwork [17] showed that this linear composition bound on can be reduced at the cost of slightly increasing the failure probability . In essence, this relaxation considers the expected privacy loss of composition of mechanisms instead of the worst-case privacy loss of the individual mechanisms. Dwork defines this as the advanced composition theorem, and proves that it applies to any differentially-private mechanism. Three commonly-used subsequent relaxed versions of differential privacy are Concentrated Differential Privacy [18], Zero Concentrated Differential Privacy [8], and Rényi Differential Privacy [50]. All of these directly bound the expected privacy loss instead of the worst-case privacy loss for composition of multiple mechanisms in order to achieve better utility. However, it is important to consider the actual impact these relaxations have on the privacy leakage, which is a main focus of this paper.
Dwork et al. [18] note that composition of multiple differential private mechanisms result in expected privacy loss which follows a subgaussian distribution. Thus, the expected privacy loss can be directly bounded by controlling the mean and variance of the subgaussian distribution. This reduces the noise that must be added to the individual mechanisms, thereby improving their utility. The authors term this as concentrated differential privacy [18]:
Definition 2.3 (Concentrated Differential Privacy (CDP)).
A randomized algorithm is -concentrated differentially-private if for all pairs of adjacent data sets and ,
where the subgaussian divergence, , is defined such that the expected privacy loss is bounded by
and the standard deviation of the centered subgaussian distribution is bounded by
. Any -DP algorithm satisfies -CDP, however the converse is not true.A variation on CDP, zero concentrated differential privacy (zCDP) [8] considers that the expected privacy loss is tightly centered around zero mean:
Definition 2.4 (Zero Concentrated Differential Privacy (zCDP)).
A randomized mechanism is -zero-concentrated differentially private if, for all neighbouring data sets and and all ,
where is the -Rényi divergence between the distribution of and the distribution of .
If satisfies -DP, then it also satisfies -zCDP. Further, if provides -zCDP, it is -DP for any . The Rényi divergence allows zCDP to be mapped back to DP, which is not the case for CDP. However, Bun and Steinke [8] give a relationship between CDP and zCDP, which allows an indirect mapping from CDP to DP (Table 1).
Definition 2.5 (Rényi Dfferential Privacy (RDP) [50]).
A randomized mechanism is said to have -Rényi differential privacy of order (which can be abbreviated as -RDP), if for any adjacent data sets , it holds that
The main difference is that CDP and zCDP require a linear bound on all
positive moments of privacy loss, whereas RDP only requires bounding one moment at a time, which allows for a more accurate numerical analysis of privacy loss. If
is an -RDP mechanism, it also satisfies -DP for any .Table 1 gives a comparison of the relaxed variations of differential privacy. For all the variations, the privacy budget grows sub-linearly with the number of compositions . For group privacy, the privacy budget for CDP and zCDP grows linearly with the group size , whereas it grows exponentially for RDP. As expected, these relaxations leak more information than pure differential privacy, which is evident from the fact that they cannot be mapped to pure -DP for any value of . In other words, there are no worst-case bounds.
Moments Accountant. Motivated by relaxations of differential privacy, Abadi et al. [1] propose the moments accountant
(MA) mechanism for bounding the expected privacy loss of differentially-private algorithms. The moments accountant tries to bound the higher order moments of the privacy loss random variable. Though the authors do not formalize this as a relaxed definition, we note that their moments bound is analogous to the Rényi divergence. In fact, the moments accountant can be considered as an instantiation of Rényi differential privacy. The moments accountant is widely used for differentially-private deep learning due to its availability as a practical framework, which we discuss in Section
2.3.2.2 Differential Privacy Methods for ML
In this section, we discuss methods for modifying machine learning algorithms to satisfy differential privacy. We first review the basics of convex optimization problems, such as empirical risk minimization (ERM) algorithms, and show the different methods of achieving differential privacy during the learning process. Next, we briefly discuss the differential privacy methods that are applicable to non-convex optimization problems, including deep learning.
ERM. Given a training data set , where is a feature matrix and is the vector of class labels, an ERM algorithm aims to reduce the convex objective function of the form,
where
is the convex loss function (such as mean square error (MSE) or cross-entropy loss) that measures the training loss for a given
and is a regularization function. Some common examples of regularization functions are penalty, which makes the vector sparse, and penalty, which shrinks the values of vector.The goal of the algorithm is to find the optimal that minimizes the objective function: . While many first order [14, 78, 37, 58] and second order [43, 40] methods exist to solve this minimization problem, the most basic procedure is gradient descent where we iteratively calculate the gradient of with respect to and update with the gradient information. This process is repeated until or some other termination condition is met.
There are three possible places for adding privacy-preserving noise during this training process, demarcated in Algorithm 1. First, we could add noise to the objective function , which gives us the objective perturbation mechanism (#1 in Algorithm 1). Second, we could add noise to the gradients at each iteration, which gives us the gradient perturbation mechanism (#2). Finally, we can add noise to obtained after the training, which gives us the output perturbation mechanism (#3). While there are other methods of achieving differential privacy such as input perturbation [15], sample-aggregate framework [52], exponential mechanism [48] and teacher ensemble framework [53], we limit our experimental analysis to gradient perturbation since our main objective is to evaluate the relaxed notions of differential privacy which are applicable to gradient perturbation mechanism.
The amount of noise to be added depends on the sensitivity of the machine learning algorithm that determines the noise needed for different DP definitions. For instance, consider logistic regression with regularization penalty. The objective function is of the form:
Assume that the training features are bounded, and . Chaudhuri et al. [11] prove that for this setting, objective perturbation requires sampling noise in the scale of , and output perturbation requires sampling noise in the scale of . The gradient of the objective function is:
which has a sensitivity of . Thus gradient perturbation requires sampling noise in the scale of at each iteration.
Deep learning. Deep learning follows the same learning procedure as in Algorithm 1, but the objective function is non-convex. As a result, the sensitivity analysis methods of Chaudhuri et al. [11] do not hold for deep learning as they require strong convexity assumption. Hence, their output and objective perturbation methods are not applicable. An alternative approach is to replace the non-convex function with a convex polynomial function [56, 57], and then use the standard objective perturbation. Application of gradient perturbation requires a bound on the gradient norm. Since the gradient norm can be unbounded in deep learning, gradient perturbation can be used after manually clipping the gradients at each iteration. As noted by Abadi et al. [1], norm clipping provides a sensitivity bound on the gradients which is required for generating noise in gradient perturbation.
2.3 Implementing Differential Privacy
Perturbation | Data Set | ||||
Chaudhuri et al. [11] | Output and Objective | Adult | 45,220 | 105 | 0.2 |
KDDCup99 | 70,000 | 119 | 0.2 | ||
Pathak et al. [55] | Output | Adult | 45,220 | 105 | 0.2 |
Hamm et al. [26] | Output | KDDCup99 | 493,000 | 123 | 1 |
URL | 200,000 | 50 | 1 | ||
Zhang et al. [80] | Objective | US | 370,000 | 14 | 0.8 |
Brazil | 190,000 | 14 | 0.8 | ||
Jain and Thakurta [34] | Objective | CoverType | 500,000 | 54 | 0.5 |
KDDCup2010 | 20,000 | 2M | 0.5 | ||
Jain and Thakurta [35] | Output and Objective | URL | 100,000 | 20M | 0.1 |
COD-RNA | 60,000 | 8 | 0.1 | ||
Song et al. [64] | Gradient | KDDCup99 | 50,000 | 9 | 1 |
MNIST | 60,000 | 15 | 1 | ||
Wu et al. [72] | Output | Protein | 72,876 | 74 | 0.05 |
CoverType | 498,010 | 54 | 0.05 | ||
Jayaraman et al. [36] | Output | Adult | 45,220 | 104 | 0.5 |
KDDCup99 | 70,000 | 122 | 0.5 |
While MNIST is normally a 10-class task, Song et al. [64] use this for ‘1 vs rest’ binary classification.
This section surveys how differential privacy has been used in machine learning applications, with a particular focus on the compromises implementers have made between privacy and utility. While the effective privacy provided by a differential privacy mechanisms depends crucially on the choice of privacy budget , setting the value is still left open to interpretations and higher privacy budgets sacrifice privacy to provide greater utility. Some of the early data analytics works on frequent pattern mining [6, 41]
[22], private record linkage [31] and recommender systems [47] were able to achieve both high utility and privacy with settings close to 1. These methods rely on finding frequency counts as a sub-routine, and hence provide -differential privacy by either perturbing the counts using Laplace noise or by releasing the top frequency counts using the exponential mechanism [48]. Machine learning, on the other hand, performs much more complex data analysis, and hence requires higher privacy budgets to maintain utility. We categorize the works in this space based on their choice of and the type of DP relaxation they use. First, we cover simple binary classification works that use small privacy budgets (). Then we survey complex classification tasks which seem to require large privacy budgets. Finally, we summarize recent works that aim to perform complex tasks with low privacy budgets by using relaxed privacy notions.Binary classification. The first practical implementation of a private machine learning algorithm was proposed by Chaudhuri and Monteleoni [10]. They provide a novel sensitivity analysis under strong convexity constraints, allowing them to use output and objective perturbation for binary logistic regression. Chaudhuri et al. [11] subsequently generalized this method for ERM algorithms. This sensitivity analysis method has since been used by many works for binary classification tasks under different learning settings (listed in Table 2). While these applications require small privacy budgets (), they only focus on learning in restricted settings such as learning with low dimensional data, smooth objective functions and strong convexity assumptions, and are only applicable to simple binary classification tasks.
There has also been a considerable progress in generalizing privacy-preserving machine learning to more complex scenarios such as learning in high dimensional settings [34, 35, 65], learning without strong convexity assumptions [66], or relaxing the assumptions on data and objective functions [63, 79, 70]. However, these advances are mainly of theoretical interest and only a few works provide practical instantiations of their proposed method [34, 35].
Task | Perturbation | Data Set | |||||
Jain et al. [33] | Online ERM | Objective | Year | 500,000 | 90 | 2 | 10 |
CoverType | 581,012 | 54 | 2 | 10 | |||
Iyengar et al. [32] | Binary ERM | Objective | Adult | 45,220 | 104 | 2 | 10 |
Binary ERM | KDDCup99 | 70,000 | 114 | 2 | 10 | ||
Multi-Class ERM | CoverType | 581,012 | 54 | 7 | 10 | ||
Multi-Class ERM | MNIST | 65,000 | 784 | 10 | 10 | ||
High Dimensional ERM | Gisette | 6,000 | 5,000 | 2 | 10 | ||
Phan et al. [56, 57] | Deep Learning | Objective | YesiWell | 254 | 30 | 2 | 1 |
MNIST | 60,000 | 784 | 10 | 1 | |||
Shokri and Shmatikov [61] | Deep Learning | Gradient | MNIST | 60,000 | 1,024 | 10 | 369,200 |
SVHN | 100,000 | 3,072 | 10 | 369,200 | |||
Zhao et al. [81] | Deep Learning | Gradient | US | 500,000 | 20 | 2 | 100 |
MNIST | 60,000 | 784 | 10 | 100 |
Complex learning tasks requiring large privacy budget. All of the above works are limited to convex learning problems with binary classification tasks. Adopting their approaches to more complex learning tasks requires higher privacy budgets (see Table 3). For instance, the online version of ERM as considered by Jain et al. [33] requires as high as 10 to achieve acceptable utility. Which means that theoretically the model’s privacy loss can be as large as . This can allow an adversary to infer the presence or absence of a record from the training set with high confidence. Further, adopting these binary classification methods for multi-class classification tasks requires even higher values. As noted by Wu et al. [72]
, it would require training a separate binary classifier for each class. Finally, high privacy budgets are required for non-convex learning algorithms, such as deep learning
[61, 81]. Since the output and objective perturbation methods of Chaudhuri et al. [11] are not applicable to non-convex settings, implementations of differentially-private deep learning rely on gradient perturbation in their iterative learning procedure. These methods do not scale to large numbers of training iterations due to the composition theorem of differential privacy which implies that the privacy budget accumulates across iterations. The only exceptions are the works of Phan et al. [56, 57] that replace the non-linear functions in deep learning with polynomial approximations and then apply objective perturbation. With this transformation, they achieve high model utility for , as shown in Table 3. However, we note that this polynomial approximation is a non-standard approach to deep learning which can limit the model’s learning capacity, and thereby affecting the model accuracy for complex tasks.Machine learning with relaxed DP definitions. To avoid the stringent composition property of differential privacy, several proposed privacy-preserving deep learning methods adopt relaxed privacy definitions based on expected privacy loss rather than worst-case loss (Section 1). Table 4 lists the gradient perturbation based works that use relaxed notions of differential to reduce the overall privacy budget during iterative learning. The utility benefit of using relaxation is evident from the fact that the privacy budget for deep learning algorithms is significantly lesser than the prior works of Shokri and Shmatikov [61] and Zhao et al. [81] which do not use any relaxation. While these relaxed definitions of differential privacy make complex iterative learning feasible for apparently reasonable values, because of the relaxed differential privacy definition, they provide no worst-case guarantees and might lead to more than expected privacy leakage in practice.
The main goal of our study is to evaluate the impact of implementation decisions regarding the privacy budget and relaxed definitions of differential privacy on the concrete privacy leakage that can be exploited by an attacker in practice. We do this by experimenting with various inference attacks, described in the next section.
Perturbation | DP Relaxation | Data Set | |||||
Huang et al. [29] | ERM | MA | Adult | 21000 | 14 | 2 | 0.5 |
Jayaraman et al. [36] | ERM | zCDP | Adult | 45,220 | 104 | 2 | 0.5 |
KDDCup99 | 70,000 | 122 | 2 | 0.5 | |||
Park et al. [54] | ERM | zCDP and MA | Stroke | 50,345 | 100 | 2 | 0.5 |
LifeScience | 26,733 | 10 | 2 | 2 | |||
Gowalla | 1,256,384 | 2 | 2 | 0.01 | |||
OlivettiFace | 400 | 4,096 | 2 | 0.3 | |||
Lee [39] | ERM | zCDP | Adult | 48,842 | 124 | 2 | 1.6 |
US | 40,000 | 58 | 2 | 1.6 | |||
Brazil | 38,000 | 53 | 2 | 1.6 | |||
Geumlek et al. [24] | ERM | RDP | Abalone | 2,784 | 9 | 2 | 1 |
Adult | 32,561 | 100 | 2 | 0.05 | |||
MNIST | 7,988 | 784 | 2 | 0.14 | |||
Beaulieu et al. [5] | Deep Learning | MA | eICU | 4,328 | 11 | 2 | 3.84 |
TCGA | 994 | 500 | 2 | 6.11 | |||
Abadi et al. [1] | Deep Learning | MA | MNIST | 60,000 | 784 | 10 | 2 |
CIFAR | 60,000 | 3,072 | 10 | 8 | |||
Yu et al. [77] | Deep Learning | MA | MNIST | 60,000 | 784 | 10 | 21.5 |
CIFAR | 60,000 | 3,072 | 10 | 21.5 | |||
Papernot et al. [53] | Deep Learning | MA | MNIST | 60,000 | 784 | 10 | 2 |
SVHN | 60,000 | 3,072 | 10 | 8 | |||
Geyer et al. [25] | Deep Learning | MA | MNIST | 60,000 | 784 | 10 | 8 |
Bhowmick et al. [7] | Deep Learning | MA | MNIST | 60,000 | 784 | 10 | 3 |
CIFAR | 60,000 | 3,072 | 10 | 3 | |||
Hynes et al. [30] | Deep Learning | MA | CIFAR | 50,000 | 3,072 | 10 | 4 |
3 Inference Attacks on Machine Learning
This section surveys the two types of inference attacks, membership inference (Section 3.1) and attribute inference (Section 3.2), and explains why they are a suitable metric for evaluating privacy leakage. Section 3.3 briefly summarizes other relevant privacy attacks on machine learning.
3.1 Membership Inference
The aim of a membership inference attack is to infer whether or not a given record is present in the training set. Membership inference attacks can uncover highly sensitive information from training data. An early membership inference attack showed that it is possible to identify individuals contributing DNA to studies that analyze a mixture of DNA from many individuals, using a statistical distance measure to determine if a known individual is in the mixture [28].
Membership inference attacks can either be completely black-box where an attacker only has query access to the target model [62], or can assume that the attacker has full white-box access to the target model, along with some auxillary information [76]. The first membership inference attack on machine learning was proposed by Shokri et al. [62]. They consider an attacker who can query the target model in a black-box way to obtain confidence scores for the queried input. The attacker tries to exploit the confidence score to determine whether the query input was present in the training data. Their attack method involves first training shadow models on a labelled data set, which can be generated either via black-box queries to the target model or through assumptions about the underlying distribution of training set. The attacker then trains an attack model using the shadow models to distinguish whether or not an input record is in the shadow training set. Finally, the attacker makes API calls to the target model to obtain confidence scores for each given input record and infers whether or not the input was part of the target model’s training set. The inference model distinguishes between the target model’s predictions for inputs that are in its training set and those it did not train on. The key assumption is that the confidence score of the target model is higher for the training instances than it would be for arbitrary instances not present in the training set. This can be due to the generalization gap, which is prominent in models that overfit to training data.
A more targeted approach was proposed by Long et al. [44] where the shadow models are trained with and without a targeted input record . At inference time, the attacker can check if the input record was present in the training set of target model. This approach tests the membership of a specific record more accurately than Shokri et al.’s approach [62]. Recently, Salem et al. [60] proposed more generic membership inference attacks by relaxing the requirements of Shokri et al. [62]. In particular, requirements on the number of shadow models, knowledge of training data distribution and the target model architecture can be relaxed without substantially degrading the effectiveness of the attack.
Yeom et al. [76] recently proposed a more computationally efficient membership inference attack when the attacker has access to the target model and knows the average training loss of the model. To test the membership of an input record, the attacker evaluates the loss of the model on the input record and then classifies it as a member if the loss is smaller than the average training loss.
Connection to Differential Privacy. Differential privacy, by definition, aims to obfuscate the presence or absence of a record in the data set. On the other hand, membership inference attacks aim to identify the presence or absence of a record in the data set. Thus, intuitively these two notions are competing each other. Li et al. [42] point to this fact and provide a direct relationship between differential privacy and membership inference attacks. Moreover, Backes et al. [3] studied membership inference attacks on microRNA studies and showed that differential privacy can reduce the success of membership inference attacks, but at the cost of utility.
Yeom et al. [76] formally define a membership inference attack as an adversarial game where a data element is selected from the distribution, which is randomly either included in the training set or not. Then, an adversary with access to the trained model attempts to determine if that element was used in training. The membership advantage is defined as the difference between the adversary’s true and false positive rates for this game. The authors prove that if the learning algorithm satisfies -differential privacy, then the adversary’s advantage is bounded by . Hence, it is natural to use membership inference attacks as a metric to evaluate the privacy leakage of differentially-private algorithms.
3.2 Attribute Inference
The aim of an attribute inference attack (also called model inversion) is to learn hidden sensitive attributes of a test input given at least API access to the model and information about the non-sensitive attributes. Fredrikson et al. [21]
formalize this attack in terms of maximizing the posterior probability estimate of the sensitive attribute. More concretely, for a test record
where the attacker knows the values of its non-sensitive attributesand all the prior probabilities of the attributes, the attacker obtains the output of the model,
, and attempts to recover the value of the sensitive attribute . The attacker essentially searches for the value of that maximizes the posterior probability . The success of this attack is based on the correlation between the sensitive attribute, , and the model output, .Yeom et al. [76] also propose an attribute inference attack using the same principle they use for their membership inference attack. The attacker evaluates the model’s empirical loss on the input instance for different values of the sensitive attribute, and reports the value which has the maximum posterior probability of achieving the empirical loss. The authors define the attribute advantage similarly to their definition of membership advantage for membership inference.
Fredrikson et al. [21] demonstrated attribute inference attacks that could identifying genetic markers based on warfarin dosage output by a model with just black-box access to model API.111This application has stirred some controversy based on the warfarin dosage output by the model itself being sensitive information correlated to the sensitive genetic markers, hence the assumption on attacker’s prior knowledge of warfarin dosage is somewhat unrealistic [46]. With additional access to confidence scores of the model (noted as white-box information by Wu et al. [71]), more complex tasks have been performed, such as recovering faces from the training data [20].
Connection to Differential Privacy. Differential privacy is mainly tailored to obfuscate the presence or absence of a record in a data set, by limiting the effect of any single record on the output of differential private model trained on the data set. Logically this definition also extends to attributes or features of a record. In other words, by adding sufficient differential privacy noise, we should be able to limit the effect of a sensitive attribute on the model’s output. This relationship between records and attributes is discussed by Yeom et al. [76]. Hence, we include these attacks in our experiments.
3.3 Other Attacks on Machine Learning
Apart from inference attacks, many other attacks have been proposed in the literature which try to infer specific information from the target model. These attacks include model stealing, hyperparameter stealing, property inference and memorization. Among these, memorization is the closest to membership inference attack. It tries to exploit the ability of high capacity models, such as deep learning models, to memorize certain sensitive patterns in the training data [9]. However, these attacks are shown to be easily thwarted by differential privacy mechanisms even with very small noise ().
Model stealing attacks aim to recover the model parameters via black-box access to the target model, either by adversarial learning [45] or by equation solving attacks [68]. Hyperparameter stealing attacks try to recover the underlying hyperparameters used during the model training, such as regularization coefficient [69] or model architecture [74]. These hyperparameters are intellectual property of commercial organizations that deploy machine learning models as a service, and hence these attacks are regarded as a threat to valuable intellectual property. A property inference attack tries to infer whether the training data set has a specific property, given a white-box access to the trained model. For instance, given access to a speech recognition model, an attacker can infer if the training data set contains speakers with a certain accent. Here the attacker can use the shadow training method of Shokri et al. [62] for distinguishing the presence and absence of a target property. These attacks have been performed on HMM and SVM models [2] and neural networks [23].
Though all these attacks may leak sensitive information about the target model or training data, the information the leak tends to be application-specific and is not clearly defined in a general way. For example, a property inference attack leaks some statistical property of the training data that is surprising to the model developer; but, the overall purpose of the model is to learn statistical properties from the training data, so there is no general definition of a property inference attack without a prescriptive decision about which statistical properties of the training data should be captured by the model and which are sensitive to leak. In addition, the attacks mentioned in this section do not closely follow the threat model of differential privacy. Thus, we only consider inference attacks for our experimental evaluation. In addition to these attacks, several poisoning and adversarial training attacks have been proposed [73, 51, 4, 75] which require an adversary that can actively interfere with the model training process. We consider these out of scope for this paper, and assume a clean training process not under the control of the adversary.
4 Empirical Evaluation
To quantify the privacy leakage of the differentially-private implementations for machine learning, we conduct experiments to measure how much an adversary can infer from a model. While differential privacy can mitigate most privacy attacks (except for property inference attacks), as motivated in Section 3, we measure privacy leakage using membership and attribute inference in our experiments. Note, however, that the conclusions we can draw from experiments like this are limited to showing a lower bound on the information leakage since they are measuring the effectiveness of a particular implementation of an attack. Experimental results from particular attacks cannot be used to make strong claims about what the best possible attack would be able to infer, especially in cases where an adversary has auxiliary information to help guide the attack. Results from our experiments, however, do provide clear evidence for when privacy protections do not appear to improve privacy in a meaningful way, and, in particular, where the expected privacy claims of relaxed mechanisms do not hold up in practice.
4.1 Experimental Setup
We evaluate the privacy leakage of two differentially-private algorithms using gradient perturbation: logistic regression for empirical risk minimization (Section 4.2) and neural networks for non-convex learning (Section 4.3). For both, we consider the different relaxed notions of differential privacy and compare their privacy leakage. The variations that we implement are naïve composition, advanced composition, zero concentrated differential privacy (zCDP) and Rényi differential privacy (RDP) (see Section 1 for details). We do not include CDP as it has the same composition property as zCDP (Table 1). For RDP, we use , using -Rényi divergence to bound the privacy loss.
We evaluate each most on two main metrics: accuracy loss, where we measure the model’s accuracy loss on test set with respect to the non-private baseline, and privacy leakage, where we measure the attacker’s advantage as defined by Yeom et al. [76]. This corresponds to the difference between the true positive rate (TPR) and the false positive rate (FPR) of the inference attack. To better understand the potential impact of leakage, we also conduct experiments to measure the actual number of members who are at risk for disclosure in a membership inference attack.
Data sets. We evaluate our models over two datasets for multi-class classification tasks: CIFAR-100 [38] and Purchase-100 [12]. CIFAR-100 consists of images of 100 real world objects, with 500 instances of each object class. We use PCA to reduce the dimensionality of records to 50. The Purchase-100 data set consists of 200,000 customer purchase records of size 100 each (corresponding to the 100 most frequently-purchased items) where the records are grouped into 100 classes based on the customers’ purchase style. For both data sets, we use 10,000 randomly-selected instances for training and 10,000 randomly-selected non-training instances for the test set. The remaining records are used for training shadow models and inference model.
Attacks. For our experiments, we use the attack frameworks of Shokri et al. [62] and Yeom et al. [76] for membership inference and the method proposed by Yeom et al. [76] for attribute inference. In Shokri et al.’s framework [62], multiple shadow models are trained on data that is sampled from the same distribution as the private data set. These shadow models are used to train an inference model to identify whether an input record belongs to the private data set. The inference model is trained using a set of records used to train the shadow models, a set of records randomly selected from the distribution that are not part of the shadow model training, along with the confidence scores output by the shadow models for all of the input records. Using these inputs, the inference model learns to distinguish the training records from the non-training records. At the inference stage, the inference model takes an input record along with the confidence score of the target model on the input record, and outputs whether the input record belongs to the target model’s private training data set. The intuition is that if the target model overfits on its training set, its confidence score for a training record will be higher than its confidence score for an otherwise similar input that was not used in training. The inference model tries to exploit this property. In our instantiation of the attack framework, we use five shadow models which all have the same model architecture as the target model. Our inference model is a neural network with two hidden layers of size 64. This setting is in accordance to the original work [62].
The attack framework of Yeom et al. [76] is simpler than Shokri et al.’s design. It assumes a white-box attacker with access to the target model’s expected training loss on the private data set, in addition to having access to the target model. For membership inference, the attacker simply observes the target model’s loss on the input record. The attacker classifies the record as a member of private data set if the loss is smaller than the target model’s expected training loss, otherwise the record is classified as a non-member. The same principle is used for attribute inference. Given an input record, the attacker brute-forces all possible values for the unknown private attribute and observes the target model’s loss, outputting the value for which the loss is closest to the target’s expected training loss. Since there are no attributes in our data sets that are explicitly annotated as private, we randomly choose five attributes, and perform the attribute inference attack on each attribute independently, and report the averaged results.
4.2 Logistic Regression Results
For both the CIFAR-100 and Purchase-100 data sets, we train a regularized logistic regression model with regularization for strong convexity. First, we train a non-private model and perform a grid search over the regularization coefficient to find the value that minimizes the classification error on test set. Next, we fix this setting to train differentially-private models using gradient perturbation. We vary between 0.5 to 1000, and report the accuracy loss and privacy leakage
. Due to the random noise addition, all the experiments are repeated five times and the average results and standard errors are reported.
We note that our gradient perturbation implementations require higher privacy budgets for achieving model accuracy compared to what was reported by Abadi et al. [1] and Yu et al. [77]. While their implementations achieve model accuracy close to non-private baselines for , our models require . The reason for this is because we do not assume pre-trained model parameters, unlike the prior works. We performed some initial experiments on Purchase-100 with the setting of Abadi et al. [1]
where we pre-train our model on a separate hold-out data set, and then privately train on the sensitive data set with gradient clipping threshold of
(Abadi et al. use ). While smaller value of corresponds to smaller noise added, it also limits the information in the gradients leading to poor or no training. Without the pre-training, we observed that the private training at achieves accuracy of only 0.033 on test set. Whereas with pre-training, the model achieves test accuracy of 0.736 after private training. The model achieves test accuracy of 0.738 by just training on the hold-out set. So, it is evident that the model is not learning anything from the private data set, and the model performance is solely due to the pre-training on the hold-out set. Hence, we require the usage of large values of to perform any useful learning on private data. For our experiments, we scale with to avoid separate hyperparameter tuning. However, in practice it is recommended to either fix based on the average gradient norm for the data set, or to do extensive hyperparameter tuning. Both the approaches would require allocating part of the privacy budget for hyperparameter tuning to ensure differential privacy.CIFAR-100 results. The baseline non-private logistic regression model achieves accuracy of 0.207 on training set and 0.150 on test set, which is competitive to the state-of-art neural network model [62] that achieves test accuracy close to 0.20 on CIFAR-100 after training on larger data set. Thus there is a small generalization gap of 0.057, which the inference attacks try to exploit. Figure 1 compares the accuracy loss for logistic regression models trained with different relaxed notions of differential privacy as we varying the privacy budget . The accuracy loss is normalized with respect to the accuracy of non-private model to clearly depict the model utility. An accuracy loss value of 1 means that the model has 100% loss and hence has no utility, whereas the value of 0 means that the model achieves same accuracy as the non-private baseline. As depicted in the figure, none of the variants learn anything useful for . For , RDP and zCDP achieve lower accuracy loss than the other methods, as expected since the relaxed definitions allow for less added noise. Notice that for , all the models achieve accuracy close to 0.01 which is random guessing for 100 class classification. For this example, there is no choice of available that provides any effective privacy for a model that does better than random guessing.



Figures 2 and 3 show the privacy leakage due to membership inference attacks on logistic regression models. We provide the attacker with a set of records half of which are from the training set. We call records in the training set members, and the remaining records non-members. The task of the attacker is to predict whether or not a given input record belongs to the training set (i.e., if it is a member). Figure 2 shows results for the black-box attacker of Shokri et al. [62], which has access to the target model’s confidence scores on the input record. For all of the variations of differential privacy, the privacy leakage for is minimal and similar (attacker advantage is close to 0.01 for ). For , naïve composition has leakage of , whereas the relaxed variants have average leakage close to .
Naive Composition | Advanced Composition | zCDP | RDP | |||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Loss | 1% | 2% | 5% | Loss | 1% | 2% | 5% | Loss | 1% | 2% | 5% | Loss | 1% | 2% | 5% | |
0.1 | 0.94 | 0 | 0 | 0 | 0.93 | 0 | 0 | 0 | 0.94 | 0 | 0 | 0 | 0.94 | 0 | 0 | 0 |
0.5 | 0.93 | 0 | 0 | 0 | 0.94 | 0 | 0 | 0 | 0.93 | 0 | 0 | 0 | 0.93 | 0 | 0 | 1 |
1.0 | 0.94 | 0 | 0 | 0 | 0.93 | 0 | 0 | 0 | 0.92 | 0 | 0 | 0 | 0.93 | 0 | 0 | 0 |
5.0 | 0.94 | 0 | 0 | 0 | 0.92 | 0 | 0 | 0 | 0.91 | 0 | 0 | 1 | 0.92 | 0 | 0 | 0 |
10.0 | 0.93 | 0 | 0 | 0 | 0.92 | 0 | 0 | 0 | 0.90 | 0 | 0 | 0 | 0.89 | 0 | 0 | 0 |
50.0 | 0.92 | 0 | 0 | 0 | 0.81 | 0 | 0 | 1 | 0.65 | 6 | 12 | 41 | 0.66 | 4 | 7 | 35 |
100.0 | 0.89 | 0 | 0 | 0 | 0.62 | 1 | 3 | 25 | 0.43 | 28 | 67 | 175 | 0.47 | 19 | 48 | 138 |
500.0 | 0.30 | 23 | 45 | 169 | 0.07 | 103 | 243 | 606 | 0.06 | 109 | 263 | 653 | 0.06 | 101 | 249 | 620 |
1000.0 | 0.11 | 54 | 103 | 346 | 0.04 | 106 | 251 | 616 | 0.04 | 115 | 279 | 676 | 0.04 | 105 | 259 | 616 |
Figure 3 shows results for the white-box attacker of Yeom et al. [62], which has access to the target model’s loss on the input record. The difference between the relaxed variants is more apparent for this attack compare to the membership inference attack in Figure 2. This is because the attacker in Shokri et al. [62] is restricted by the attack model’s learning capacity. As expected, zCDP and RDP relaxations leak the most, followed by advanced composition. Naïve composition does not have any leakage for , but the leakage reaches for . The observed leakage of all the variations is in accordance with the noise magnitude required for different differential privacy guarantees.
Figure 4 depicts the privacy leakage due to the attribute inference attack. As shown, the privacy leakage of zCDP is highest, closely followed by RDP. The relaxed variants have privacy leakage comparable to naïve composition for , and tend to leak more for larger values of . Naïve composition has low privacy leakage for (attacker advantage of at ), but it quickly increases to for . But for meaningful privacy budgets there is no significant leakage () for any of the methods. From Figures 1–4, we see as expected that as privacy budgets increase, the attacker’s advantage (privacy leakage) increases and the model utility increases (accuracy loss decreases). This trend is consistent across all variations.

To gain more understanding of the impact of privacy leakage, we investigate the actual number of training set members exposed to the attacker for different differential privacy variations. We assume the attacker has some limited tolerance for falsely exposing a member (that is, a bound on the acceptable false positive rate), and sets the required threshold score for the inference model output as the level needed to achieve that false positive rate. Then, we count the number of members in the private training data set for whom the inference model output exceeds that confidence threshold. Table 5 summarizes the results, reporting the number of members exposed to an adversary who tolerates false positive rates of 1%, 2%, and 5%. As we increase the tolerance threshold, there is a gradual increase in membership leakage for all the methods. While all the methods are resistant to attack for , the leakage of relaxed variants increases drastically for higher values.
Purchase-100 results. The baseline non-private logistic regression model achieves accuracy of 0.8127 on the training set and 0.6322 on test set. In comparison, Google ML platform’s black-box trained model achieves a test accuracy of 0.656 for Purchase-100 (see Shokri et al. [62] for details). Since the results are similar to the CIFAR-100 data set, we only show the plots comparing the accuracy loss and privacy leakage of models against the membership inference attack of Yeom et al. [76] for this data set.
Figure 5 shows the accuracy loss of all variants on Purchase-100 data set. None of the variants have any utility for . For , naïve composition achieves accuracy loss of , while all the other variants achieve accuracy loss close to 0.2 (zCDP achieves the least loss of ). None of the variants achieve model utility close to the non-private baseline for any privacy budget.

Figure 6 shows the privacy leakage comparison of the variants against membership inference attack on Purchase-100 data set. The leakage is in accordance to the noise each variant adds and it increases proportional to the model utility. Hence, if a model has reasonable utility, it is bound to leak membership information. Table 7 (in the Appendix) shows the number of individual members exposed, with similar results to the findings for CIFAR-100.

4.3 Neural Networks
We train a neural network model consisting of two hidden layers and an output layer. The hidden layers have 256 neurons that use
ReLU activation. The output layer is a softmax layer with 100 neurons, each corresponding to a class label. This architecture is similar to the one used by Shokri et al. [62].CIFAR-100 results. The baseline non-private neural network model achieves accuracy of 0.9856 on the training set and 0.1788 on test set, which is competitive to the neural network model of Shokri et al. [62]. Their model is trained on training set of size 29,540 and achieves test accuracy of 0.20, whereas our model is trained on 10,000 training instances. Thus, there is a huge generalization gap of 0.8068, which the inference attacks can exploit. Figure 7 shows the accuracy loss comparison of neural network models trained with different relaxed notions of differential privacy with varying privacy budget . While the relaxed variants begin gaining accuracy for , the model trained with naïve composition does not learning anything useful until , at which point the other variants achieve accuracy loss close to zero.

Figure 8 shows the privacy leakage due to membership inference attacks on neural network models trained with different relaxed notions for both attacks. The privacy leakage for all the variations of differential privacy is close to zero for . For , with the Shokri et al. attack, naïve composition has leakage of compared to for advanced composition, for zCDP, and for RDP. For the white-box attacker of Yeom et al. [76], the zCDP and RDP relaxations begin leaking privacy at whereas naïve composition does not leak for any value of . RDP leaks the most for (membership advantage of ). This is because these relaxed variations consider only expected privacy loss, and hence add considerably less noise in comparison to naïve composition. Naïve composition achieves strong privacy against membership inference attackers, but fails to learning anything useful. No option appears to provide both reasonable model utility and meaningful privacy.
Naive Composition | Advanced Composition | zCDP | RDP | |||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Loss | 1% | 2% | 5% | Loss | 1% | 2% | 5% | Loss | 1% | 2% | 5% | Loss | 1% | 2% | 5% | |
0.1 | 0.95 | 0 | 0 | 0 | 0.95 | 0 | 0 | 0 | 0.94 | 0 | 0 | 0 | 0.93 | 0 | 0 | 0 |
0.5 | 0.94 | 0 | 0 | 0 | 0.94 | 0 | 0 | 0 | 0.93 | 0 | 0 | 0 | 0.93 | 0 | 5 | 90 |
1.0 | 0.94 | 0 | 0 | 0 | 0.94 | 0 | 0 | 0 | 0.92 | 0 | 0 | 6 | 0.91 | 0 | 4 | 94 |
5.0 | 0.94 | 0 | 0 | 0 | 0.93 | 0 | 0 | 0 | 0.83 | 0 | 3 | 16 | 0.83 | 0 | 0 | 45 |
10.0 | 0.94 | 0 | 0 | 0 | 0.87 | 0 | 0 | 1 | 0.81 | 0 | 0 | 20 | 0.80 | 0 | 11 | 109 |
50.0 | 0.95 | 0 | 0 | 0 | 0.73 | 0 | 0 | 12 | 0.64 | 0 | 1 | 87 | 0.64 | 0 | 1 | 70 |
100.0 | 0.93 | 0 | 0 | 0 | 0.61 | 1 | 8 | 32 | 0.49 | 30 | 75 | 281 | 0.48 | 11 | 43 | 202 |
500.0 | 0.93 | 0 | 0 | 2 | 0.06 | 26 | 76 | 331 | 0.00 | 54 | 119 | 399 | 0.00 | 40 | 104 | 379 |
1000.0 | 0.59 | 0 | 0 | 11 | 0.06 | 13 | 53 | 359 | 0.00 | 28 | 78 | 416 | 0.07 | 22 | 64 | 383 |
Non-private model leaks 155, 425 and 2667 members for 1%, 2% and 5% FPR respectively.


Figure 9 depicts the privacy leakage due to attribute inference attack on neural network models trained with different relaxed notions. Again zCDP and RDP are more vulnerable to attribute inference attack due to their relaxation of differential privacy guarantee. Naïve composition is resistant to the attack for . In comparison to the case of logistic regression, here the relaxed variants have more privacy leakage as the generalization gap for neural network is large.
As for logistic regression, we further investigate the actual number of training set members exposed to the attacker and report the results in Table 6. The impact ot the increased leakage of relaxed definitions is much more severe for neural networks, revealing members at the 2% false positive threshold for RDP even at . Naïve composition is secure even with high privacy budgets, revealing no members at 2% false positive threshold for any value, even though the non-private model leaks between 155 members at 1% false positive threshold. On the other hand, the relaxed variants of differential privacy expose many members, even with fairly low values, and expose hundreds of members at the privacy budgets needed to achieve low accuracy loss. As shown, zCDP leaks more members than RDP for . This is because RDP adds more noise when . Since we fix , RDP perturbs more than zCDP for .

Purchase-100 results. The baseline non-private neural network model achieves accuracy of 0.9998 on the training set and 0.7438 on test set. Our model performs better than the neural network model Shokri et al. [62] trained on the same data set which reports 0.670 test accuracy. Since most of the results are similar to the results on CIFAR-100 data set, we exclude the plots on privacy leakage comparison of differential privacy variants against the attribute inference attack of Yeom et al. [76] and the membership inference attack of Shokri et al. [62]. We only show the accuracy loss (Figure 10) and privacy leakage results for membership inference attack of Yeom et al. [76]. The trends for both accuracy and privacy are similar to that of logistic regression model on Purchase-100 data set (Figure 5). However, here the relaxed variants achieve model utility close to the non-private baseline for , while naïve composition achieves accuracy loss of . Figure 11 (in the Appendix) shows the privacy leakage comparison of the variants against membership inference attack and Table 8 shows the number of individuals exposed. The results are consistent with those observed for CIFAR-100.
5 Conclusion
Differential privacy has earned a well-deserved reputation providing a principled and powerful mechanisms for ensuring provable privacy. However, when it is implemented for challenging tasks such as machine learning, compromises must be made to preserve utility. It is essential that the privacy impact of those compromises is well understood by implementers deploying differential privacy on sensitive data. Our results are a step towards improving that understanding, and reveal that the commonly used relaxations of differential privacy may provide unacceptable utility-privacy tradeoffs. We hope our study will encourage more careful assessments of the practical privacy value of formal claims based on differential privacy, and lead to work on deeper understanding of the privacy impact of design decisions when deploying differential privacy, and eventually to solutions that provide desirable, and well understood, utility-privacy tradeoffs.
Availability
Open source code for reproducing all of our experiments is available at https://github.com/bargavj/EvaluatingDPML.
Acknowledgments
This work was partially funded by a grant from the National Science Foundation (#1717950). We would like to thank Atallah Hezbor, Faysal Hossain Shezan, Tanmoy Sen, Max Naylor, Joshua Holtzman and Nan Yang for helping in systematizing the related works. Finally, we would also like to thank Congzheng Song and Samuel Yeom for providing their implementation of inference attacks.
References
- [1] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In ACM Conference on Computer and Communications Security, 2016.
- [2] Giuseppe Ateniese, Luigi V Mancini, Angelo Spognardi, Antonio Villani, Domenico Vitali, and Giovanni Felici. Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers. International Journal of Security and Networks, 10, 2015.
- [3] Michael Backes, Pascal Berrang, Mathias Humbert, and Praveen Manoharan. Membership privacy in microrna-based studies. In ACM Conference on Computer and Communications Security, 2016.
- [4] Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. CoRR, abs/1807.00459, 2018.
- [5] Brett K Beaulieu-Jones, William Yuan, Samuel G Finlayson, and Zhiwei Steven Wu. Privacy-preserving distributed deep learning for clinical data. arXiv preprint arXiv:1812.01484, 2018.
- [6] Raghav Bhaskar, Srivatsan Laxman, Adam Smith, and Abhradeep Thakurta. Discovering frequent patterns in sensitive data. In 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2010.
- [7] Abhishek Bhowmick, John Duchi, Julien Freudiger, Gaurav Kapoor, and Ryan Rogers. Protection against reconstruction and its applications in private federated learning. arXiv preprint arXiv:1812.00984, 2018.
- [8] Mark Bun and Thomas Steinke. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Theory of Cryptography Conference, 2016.
- [9] Nicholas Carlini, Chang Liu, Jernej Kos, Úlfar Erlingsson, and Dawn Song. The secret sharer: Measuring unintended neural network memorization & extracting secrets. arXiv preprint 1802.08232, 2018.
- [10] Kamalika Chaudhuri and Claire Monteleoni. Privacy-preserving logistic regression. In Advances in Neural Information Processing Systems, 2009.
- [11] Kamalika Chaudhuri, Claire Monteleoni, and Anand D. Sarwate. Differentially private empirical risk minimization. Journal of Machine Learning Research, 2011.
- [12] Kaggle Competition. Acquire valued shoppers challenge, 2014.
- [13] Zeyu Ding, Yuxin Wang, Guanhong Wang, Danfeng Zhang, and Daniel Kifer. Detecting violations of differential privacy. In ACM Conference on Computer and Communications Security, 2018.
- [14] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul), 2011.
- [15] John C Duchi, Michael I Jordan, and Martin J Wainwright. Local privacy and statistical minimax rates. In Symposium on Foundations of Computer Science. IEEE, 2013.
- [16] Cynthia Dwork. Differential Privacy: A Survey of Results. In International Conference on Theory and Applications of Models of Computation, 2008.
- [17] Cynthia Dwork and Aaron Roth. The Algorithmic Foundations of Differential Privacy. Foundations and Trends in Theoretical Computer Science, 2014.
- [18] Cynthia Dwork and Guy N. Rothblum. Concentrated differential privacy. arXiv preprint arXiv:1603.01887, 2016.
- [19] Úlfar Erlingsson, Vasyl Pihur, and Aleksandra Korolova. RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response. In ACM Conference on Computer and Communications Security, 2014.
- [20] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In ACM Conference on Computer and Communications Security, 2015.
- [21] Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ristenpart. Privacy in pharmacogenetics: An end-to-end case study of personalized warfarin dosing. In 23rd USENIX Security Symposium, 2014.
- [22] Arik Friedman and Assaf Schuster. Data mining with differential privacy. In 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2010.
- [23] Karan Ganju, Qi Wang, Wei Yang, Carl A Gunter, and Nikita Borisov. Property inference attacks on fully connected neural networks using permutation invariant representations. In ACM Conference on Computer and Communications Security, 2018.
- [24] Joseph Geumlek, Shuang Song, and Kamalika Chaudhuri. Renyi differential privacy mechanisms for posterior sampling. In Advances in Neural Information Processing Systems, 2017.
- [25] Robin C Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arXiv preprint arXiv:1712.07557, 2017.
- [26] Jihun Hamm, Paul Cao, and Mikhail Belkin. Learning privately from multiparty data. In International Conference on Machine Learning, 2016.
- [27] Michael Hay, Ashwin Machanavajjhala, Gerome Miklau, Yan Chen, and Dan Zhang. Principled evaluation of differentially private algorithms using dpbench. In ACM SIGMOD International Conference on Management of Data, 2016.
- [28] Nils Homer, Szabolcs Szelinger, Margot Redman, David Duggan, Waibhav Tembe, Jill Muehling, John V Pearson, Dietrich A Stephan, Stanley F Nelson, and David W Craig. Resolving individuals contributing trace amounts of DNA to highly complex mixtures using high-density SNP genotyping microarrays. PLoS Genetics, 4, 2008.
- [29] Zonghao Huang, Rui Hu, Yanmin Gong, and Eric Chan-Tin. Dp-admm: Admm-based distributed learning with differential privacy. arXiv preprint arXiv:1808.10101, 2018.
- [30] Nick Hynes, Raymond Cheng, and Dawn Song. Efficient deep learning on multi-source private data. arXiv preprint arXiv:1807.06689, 2018.
- [31] Ali Inan, Murat Kantarcioglu, Gabriel Ghinita, and Elisa Bertino. Private record matching using differential privacy. In 13th International Conference on Extending Database Technology, 2010.
- [32] Roger Iyengar, Joseph P Near, Dawn Song, Om Thakkar, Abhradeep Thakurta, and Lun Wang. Towards practical differentially private convex optimization. In IEEE Symposium on Security and Privacy, 2019.
- [33] Prateek Jain, Pravesh Kothari, and Abhradeep Thakurta. Differentially private online learning. In 25th Annual Conference on Learning Theory, 2012.
- [34] Prateek Jain and Abhradeep Thakurta. Differentially private learning with kernels. In International Conference on Machine Learning, 2013.
- [35] Prateek Jain and Abhradeep Guha Thakurta. (near) dimension independent risk bounds for differentially private learning. In International Conference on Machine Learning, 2014.
- [36] Bargav Jayaraman, Lingxiao Wang, David Evans, and Quanquan Gu. Distributed learning without distress: Privacy-preserving empirical risk minimization. In Advances in Neural Information Processing Systems, 2018.
- [37] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
- [38] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
-
[39]
Jaewoo Lee.
Differentially private variance reduced stochastic gradient descent.
In International Conference on New Trends in Computing Sciences (ICTCS). IEEE, 2017. - [40] Dong-Hui Li and Masao Fukushima. A modified bfgs method and its global convergence in nonconvex minimization. Journal of Computational and Applied Mathematics, 129, 2001.
- [41] Ninghui Li, Wahbeh Qardaji, Dong Su, and Jianneng Cao. Privbasis: Frequent itemset mining with differential privacy. The VLDB Journal, 5, 2012.
- [42] Ninghui Li, Wahbeh Qardaji, Dong Su, Yi Wu, and Weining Yang. Membership privacy: A unifying framework for privacy definitions. In ACM Conference on Computer and Communications Security, 2013.
- [43] Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45, 1989.
- [44] Yunhui Long, Vincent Bindschaedler, and Carl A. Gunter. Towards measuring membership privacy. arXiv preprint arXiv:1712.09136, 2017.
- [45] Daniel Lowd and Christopher Meek. Adversarial learning. In 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2005.
- [46] Frank McSherry. Statistical inference considered harmful. https://github.com/frankmcsherry/blog/blob/master/posts/2016-06-14.md, 2016.
- [47] Frank McSherry and Ilya Mironov. Differentially private recommender systems: Building privacy into the netflix prize contenders. In 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2009.
- [48] Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. In 48th Symposium on Foundations of Computer Science, 2007.
- [49] Frank D. McSherry. Privacy integrated queries: An extensible platform for privacy-preserving data analysis. In ACM SIGMOD International Conference on Management of Data, 2009.
- [50] Ilya Mironov. Renyi differential privacy. In 30th IEEE Computer Security Foundations Symposium, 2017.
-
[51]
Luis Munoz-González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin
Wongrassamee, Emil C. Lupu, and Fabio Roli.
Towards poisoning of deep learning algorithms with back-gradient
optimization.
In 10th
ACM Workshop on Artificial Intelligence and Security
, 2017. -
[52]
Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith.
Smooth sensitivity and sampling in private data analysis.
In 39th
ACM Symposium on Theory of Computing
, 2007. - [53] Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi-supervised knowledge transfer for deep learning from private training data. arXiv preprint arXiv:1610.05755, 2016.
-
[54]
Mijung Park, Jimmy Foulds, Kamalika Chaudhuri, and Max Welling.
Dp-em: Differentially private expectation maximization.
In Artificial Intelligence and Statistics, 2017. - [55] Manas Pathak, Shantanu Rane, and Bhiksha Raj. Multiparty Differential Privacy via Aggregation of Locally Trained Classifiers. In Advances in Neural Information Processing Systems, 2010.
- [56] NhatHai Phan, Yue Wang, Xintao Wu, and Dejing Dou. Differential privacy preservation for deep auto-encoders: an application of human behavior prediction. In AAAI, volume 16, 2016.
-
[57]
NhatHai Phan, Xintao Wu, and Dejing Dou.
Preserving differential privacy in convolutional deep belief networks.
Machine Learning, 106, 2017. - [58] Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30, 1992.
- [59] Md Atiqur Rahman, Tanzila Rahman, Robert Laganiere, Noman Mohammed, and Yang Wang. Membership inference attack against differentially private deep learning model. Transactions on Data Privacy, 11, 2018.
- [60] Ahmed Salem, Yang Zhang, Mathias Humbert, Mario Fritz, and Michael Backes. ML-Leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint arXiv:1806.01246, 2018.
- [61] Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. In ACM Conference on Computer and Communications Security, 2015.
- [62] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In IEEE Symposium on Security and Privacy, 2017.
-
[63]
Adam Smith and Abhradeep Thakurta.
Differentially Private Feature Selection via Stability Arguments, and the Robustness of the Lasso.
In Proceedings of Conference on Learning Theory, 2013. - [64] Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differentially private updates. In IEEE Global Conference on Signal and Information Processing (GlobalSIP), 2013.
- [65] Kunal Talwar, Abhradeep Thakurta, and Li Zhang. Private empirical risk minimization beyond the worst case: The effect of the constraint set geometry. arXiv preprint arXiv:1411.5417, 2014.
- [66] Kunal Talwar, Abhradeep Thakurta, and Li Zhang. Nearly Optimal Private LASSO. In Advances in Neural Information Processing Systems, 2015.
- [67] Jun Tang, Aleksandra Korolova, Xiaolong Bai, Xueqiang Wang, and Xiaofeng Wang. Privacy loss in Apple’s implementation of differential privacy on MacOS 10.12. arXiv preprint arXiv:1709.02753, 2017.
- [68] Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction APIs. In USENIX Security Symposium, 2016.
- [69] Binghui Wang and Neil Zhenqiang Gong. Stealing hyperparameters in machine learning. In IEEE Symposium on Security and Privacy, 2018.
- [70] Di Wang, Minwei Ye, and Jinhui Xu. Differentially private empirical risk minimization revisited: Faster and more general. In Advances in Neural Information Processing Systems. 2017.
- [71] Xi Wu, Matthew Fredrikson, Somesh Jha, and Jeffrey F Naughton. A methodology for formalizing model-inversion attacks. In 29th IEEE Computer Security Foundations Symposium, 2016.
- [72] Xi Wu, Fengan Li, Arun Kumar, Kamalika Chaudhuri, Somesh Jha, and Jeffrey Naughton. Bolt-on differential privacy for scalable stochastic gradient descent-based analytics. In ACM SIGMOD International Conference on Management of Data, 2017.
- [73] Huang Xiao, Battista Biggio, Blaine Nelson, Han Xiao, Claudia Eckert, and Fabio Roli. Support vector machines under adversarial label contamination. Neurocomputing, 160, 2015.
- [74] Mengjia Yan, Christopher Fletcher, and Josep Torrellas. Cache telepathy: Leveraging shared resource attacks to learn dnn architectures. arXiv preprint arXiv:1808.04761, 2018.
- [75] Chaofei Yang, Qing Wu, Hai Li, and Yiran Chen. Generative poisoning attack method against neural networks. CoRR, abs/1703.01340, 2017.
- [76] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 31st IEEE Computer Security Foundations Symposium, 2018.
- [77] Lei Yu, Ling Liu, Calton Pu, Mehmet Emre Gursoy, and Stacey Truex. Differentially private model publishing for deep learning. In IEEE Symposium on Security and Privacy, 2019.
- [78] Matthew D Zeiler. Adadelta: An adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.
- [79] Jiaqi Zhang, Kai Zheng, Wenlong Mou, and Liwei Wang. Efficient private erm for smooth objectives. In 26th International Joint Conference on Artificial Intelligence, 2017.
-
[80]
Jun Zhang, Zhenjie Zhang, Xiaokui Xiao, Yin Yang, and Marianne Winslett.
Functional mechanism: Regression analysis under differential privacy.
The VLDB Journal, 5, 2012. - [81] Lingchen Zhao, Yan Zhang, Qian Wang, Yanjiao Chen, Cong Wang, and Qin Zou. Privacy-preserving collaborative deep learning with irregular participants. arXiv preprint arXiv:1812.10113, 2018.
Appendix A Additional results for Purchase-100

Naïve Composition | Advanced Composition | zCDP | RDP | |||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Loss | 1% | 2% | 5% | Loss | 1% | 2% | 5% | Loss | 1% | 2% | 5% | Loss | 1% | 2% | 5% | |
0.1 | 0.99 | 0 | 0 | 0 | 0.98 | 0 | 0 | 0 | 0.98 | 0 | 0 | 0 | 0.98 | 0 | 0 | 0 |
0.5 | 0.98 | 0 | 0 | 0 | 0.99 | 0 | 0 | 0 | 0.98 | 0 | 0 | 0 | 0.98 | 0 | 0 | 0 |
1.0 | 0.99 | 0 | 0 | 0 | 0.98 | 0 | 0 | 0 | 0.98 | 0 | 0 | 0 | 0.97 | 0 | 0 | 0 |
5.0 | 0.98 | 0 | 0 | 0 | 0.97 | 0 | 0 | 8 | 0.98 | 0 | 1 | 85 | 0.97 | 1 | 5 | 101 |
10.0 | 0.98 | 0 | 0 | 0 | 0.97 | 0 | 2 | 20 | 0.97 | 0 | 2 | 101 | 0.97 | 1 | 3 | 105 |
50.0 | 0.97 | 0 | 0 | 0 | 0.96 | 0 | 1 | 39 | 0.95 | 0 | 9 | 205 | 0.95 | 4 | 12 | 138 |
100.0 | 0.97 | 0 | 0 | 0 | 0.95 | 0 | 9 | 90 | 0.92 | 5 | 30 | 210 | 0.93 | 3 | 20 | 163 |
500.0 | 0.87 | 0 | 0 | 2 | 0.60 | 37 | 76 | 216 | 0.55 | 64 | 131 | 390 | 0.58 | 49 | 84 | 268 |
1000.0 | 0.63 | 3 | 7 | 22 | 0.22 | 89 | 182 | 435 | 0.19 | 118 | 231 | 582 | 0.23 | 87 | 156 | 411 |
0.00 | 212 | 363 | 957 | 0.00 | 212 | 363 | 957 | 0.00 | 212 | 363 | 957 | 0.00 | 212 | 363 | 957 |
Naïve Composition | Advanced Composition | zCDP | RDP | |||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Loss | 1% | 2% | 5% | Loss | 1% | 2% | 5% | Loss | 1% | 2% | 5% | Loss | 1% | 2% | 5% | |
0.1 | 0.99 | 0 | 0 | 0 | 0.99 | 0 | 0 | 0 | 0.99 | 0 | 0 | 0 | 0.98 | 0 | 0 | 0 |
0.5 | 0.99 | 0 | 0 | 0 | 0.98 | 0 | 0 | 0 | 0.98 | 0 | 0 | 0 | 0.98 | 0 | 0 | 1 |
1.0 | 0.99 | 0 | 0 | 0 | 0.98 | 0 | 0 | 1 | 0.98 | 0 | 0 | 0 | 0.98 | 0 | 0 | 0 |
5.0 | 0.98 | 0 | 0 | 0 | 0.98 | 0 | 0 | 0 | 0.96 | 0 | 11 | 163 | 0.96 | 0 | 13 | 233 |
10.0 | 0.99 | 0 | 0 | 0 | 0.96 | 0 | 0 | 0 | 0.96 | 0 | 4 | 104 | 0.96 | 1 | 24 | 219 |
50.0 | 0.98 | 0 | 0 | 0 | 0.91 | 0 | 0 | 1 | 0.94 | 11 | 37 | 147 | 0.94 | 3 | 20 | 168 |
100.0 | 0.98 | 0 | 0 | 0 | 0.86 | 0 | 0 | 0 | 0.85 | 21 | 59 | 221 | 0.85 | 9 | 42 | 176 |
500.0 | 0.95 | 36 | 74 | 107 | 0.06 | 31 | 61 | 166 | 0.12 | 50 | 98 | 256 | 0.06 | 49 | 74 | 196 |
1000.0 | 0.65 | 0 | 0 | 0 | 0.02 | 0 | 0 | 217 | 0.02 | 54 | 102 | 266 | 0.02 | 0 | 0 | 198 |
0.00 | 0 | 196 | 574 | 0.00 | 0 | 196 | 574 | 0.00 | 0 | 196 | 574 | 0.00 | 0 | 196 | 574 |
First column for each method depicts the accuracy loss with respect to non-private model.