1 Introduction
With machine learning (ML) becoming ubiquitous in many aspects of our society, questions of its privacy and security take centre stage. A growing field of research in privacy attacks on ML [14, 30, 17, 34] tells us that it is possible to infer information about training data even in a blackbox setting, without access to model parameters. A wider population, however, is concerned with privacy practices used in the ML development cycle, such as company employees or contractors manually inspecting and annotating user data^{1}^{1}1https://www.theguardian.com/technology/2020/jan/10/skypeaudiogradedbyworkersinchinawithnosecuritymeasures^{2}^{2}2https://www.bloomberg.com/news/articles/20190410/isanyonelisteningtoyouonalexaaglobalteamreviewsaudio.
The problem of privacy attacks is often tackled with adding a differentially private mechanism to the model training procedure [1]. Differential privacy (DP) [13]
provides a rigorous theoretical guarantee, which states (informally) that the algorithm output would not significantly change when a single user adds or removes their data, except with small (failure) probability. Another approach gaining popularity is
federated learning (FL) [22, 7], where a central entity trains a model by computing updates locally ondevice and then securely aggregating these updates on a server. This way user data never leave their devices.In spite of significant progress, neither of these approaches solves the problem of manual data labelling. Moreover, it creates an additional hurdle for developers, as they cannot inspect data, especially in decentralised settings, making it difficult to understand the model behaviour and find bugs in data and implementations. augenstein2019generative augenstein2019generative provide a more complete characterisation of these questions.
This paper follows augenstein2019generative augenstein2019generative in adopting generative adversarial networks (GAN) [15] trained in a privacypreserving manner for addressing these issues. More specifically, we use the notion of Bayesian differential privacy (BDP) [32], which takes into account the data distribution and provides a more meaningful guarantee for indistribution samples than classical DP. Intuitively, when DP has uniform failure probability for all data points, BDP allows it to be nonuniform, thereby discounting points that are naturally difficult to hide and providing a strong guarantee for the rest of the dataset. Since both can use the same obfuscation mechanism, while computing two privacy bounds in parallel, a DP guarantee would still hold for outofdistribution samples. More details on the overall approach and privacy are provided in Section 4.
The advantage of using this privacy definition is that it enables generating data of higher fidelity, compared to previous work on GANs with DP, allowing for finergrained inspection of data. While some problems with data or data pipelines can be discovered using very coarse samples (e.g. pixel intensity inversion in [5]), more subtle bugs, like partial data corruption, would require samples of much better quality, rendering the DP guarantee too loose to be meaningful. Moreover, if fidelity is high enough, synthetic data can be used for annotation and training itself, removing the related privacy concerns and extending applicability of FL. We evaluate our solution in these two aspects in Section 5.
The main contributions of this paper are as follows:

we use Bayesian DP to enable higher quality GAN samples, while still providing a strong privacy guarantee;

we demonstrate that this technique can be used to discover finer data errors than has been previously reported;

we also show that for some tasks synthetic data are of high enough quality to be used for labelling and training.
2 Related Work
A rapidly expanding area of privacypreserving machine learning research has been recently focused on the attacks that compromise privacy of training data, such as model inversion [14] and membership inference [30]. The former is based on observing the output probabilities of the target model for a given class and performing gradient descent on an input reconstruction. The latter assumes an attacker with access to similar data, which is used to train ”shadow” models, mimicking the target, and the attack model, which predicts if a certain example has already been seen during training based on its output probabilities. Both attacks can be performed in a blackbox setting, without access to the model internal parameters.
Differential privacy (DP) [13] is widely accepted as the gold standard for preventing such attacks. One of the early takes on the problem is to use disjoint datasets and distributed training with DP. For example, [29] propose to train a model in a distributed manner by communicating sanitised updates from participants to a central authority. Such a method, however, yields high privacy losses [1, 26]. An alternative technique suggested by [26] also uses disjoint training sets and builds an ensemble of independently trained teacher models to transfer knowledge to a student model by labelling public data. This result has been extended in [27] to achieve stateoftheart image classification results in a private setting (with singledigit DP bounds). A different approach is taken by [1]
. They propose using differentially private stochastic gradient descent (DPSGD) to train deep learning models in a private manner. This approach achieves high accuracy maintaining relatively low DP bounds and being simpler to implement, but may also require pretraining on public data.
Due to the fact that the DP threat model is extremely broad, achieving a reasonable guarantee may be difficult or even impossible. For this reason, a number of alternative definitions has been proposed over the recent years, aimed at relaxing the guarantee or providing tighter composition bounds under certain assumptions. Examples are computational DP [24], mutualinformation privacy [23, 35], different versions of concentrated DP (CDP [12], zCDP [9], tCDP [8]), and RényiDP (RDP) [25]. Some other relaxations [2, 28, 10] tip the balance even further in favour of applicability at the cost of weaker guarantees, for example considering the averagecase instead of the worstcase [31].
In this work, we rely on another relaxation, called Bayesian differential privacy [32]. This notion utilises the fact that data come from a particular distribution, and not all data samples are equally likely (e.g. unlikely to find a sound record among ECG samples). At the same time, it maintains a similar probabilistic interpretation of its parameters and
. It is worth noting, that unlike some of the relaxations mentioned above, Bayesian DP can provide a tail bound on privacy loss, similarly to the moments accountant (MA)
[1], and is not limited to a particular dataset, but rather a particular type of data (e.g. emails, MRI images, etc.), which is a much more permitting assumption.Up until recently, another aspect of privacy in machine learning has been largely overlooked: the human involvement in the development cycle and manual data processing. These issues can be mitigated, at least partially, by federated learning (FL) [22], which brings a great promise for user privacy. Yet, FL paradigm creates additional problems of its own. augenstein2019generative augenstein2019generative provide a good starting point, systematising these problems and proposing a solution by the use of synthetic data. Although privacypreserving data synthesis using GANs has been introduced in earlier works [6, 37, 38, 31, 18, 21], these papers mainly focused on achieving high utility of synthetic data without addressing a broader scope of privacy leakage via manual data handling.
A common problem of privacypreserving GANs, however, is that the generated samples have very low fidelity, unless the privacy guarantee is unreasonably weak. Our approach makes progress in exactly this perspective: we can achieve much higher quality outputs with little compromise in privacy guarantees (and only for outliers that are difficult to hide). As a result, our synthetic data yield better performance of downstream analytics, and simultaneously, provide more powerful data inspection capabilities.
3 Preliminaries
In this section, we provide some background useful for understanding the paper.
We use to represent neighbouring (adjacent) datasets. If not specified, it is assumed that these datasets differ in a single example. Individual examples in a dataset are denoted by or , while the example by which two datasets differ—by . We assume , whenever possible to do so without loss of generality. The private learning outcomes (i.e. noised gradients) are denoted by .
Definition 1.
A randomised function (mechanism) with domain and range satisfies differential privacy if for any two adjacent inputs and for any set of outcomes the following holds:
(1) 
Definition 2.
Privacy loss of a randomised mechanism for inputs and outcome takes the following form:
(2) 
Definition 3.
The Gaussian noise mechanism achieving DP, for a function , is defined as
(3) 
where and is the L2sensitivity of .
For more details on differential privacy, the Gaussian mechanism, and how to use it in machine learning, we refer the reader to [11, 1].
Definition 4.
A randomised function (algorithm) with domain and range satisfies (weak) Bayesian differential privacy if for any two adjacent datasets , differing in a single data point , and for any set of outcomes the following holds:
(4) 
While the definition of BDP is very close to that of DP, there are some important differences: the interpretation of is slightly different, data are assumed to come from a distribution (although it is not required to be known), and samples are assumed to be exchangeable [3]. Nonetheless, this notion remains applicable in a wide range of practical scenarios [32].
In parts of the paper, we refer to augenstein2019generative classification of ML developer tasks, which can be condensed to:
T1  Sanity checking data.
T2  Debugging mistakes.
T3  Debugging unknown labels / classes.
T4  Debugging poor performance on certain classes / slices / users.
T5  Human labelling of examples.
T6  Detecting bias in the training data.
4 Our Approach
In this section, we describe our approach, intuition behind it, its privacy analysis, and discuss how to extend it to federated learning settings.
4.1 Intuition
The primary distinction of Bayesian differential privacy is that it takes into account the data distribution, and by extension, assumes that all data points are drawn from the same distribution, although these distributions may be multimodal, highly complex, and generally unknown. This is a natural hypothesis in many machine learning applications, but especially so when working with generative models like GANs.
The task of generative modelling in itself is to learn an underlying data distribution, and thus, a common distribution is an implicit belief. This results in an organic match with BDP, because there are no assumptions to add to the problem.
Another part of our intuition is that the foremost source of privacy leakage are outliers. On the one hand, their respective privacy loss would be discounted in BDP accounting due to their low probability. On the other hand, we can reduce the number of samples generated by the GAN to decrease the chances of these outliers appearing in the synthetic dataset.
4.2 Overview
We are given a dataset of labelled () or unlabelled () examples. This dataset can be decentralised, in which case we would use FL (see Section 4.4). Our task is to train a GAN, which consists of the generator and the critic (discriminator), to generate synthetic samples from .
Our privacy mechanism follows the previous work on privacypreserving GANs [6, 37]. More specifically, it applies the Gaussian mechanism (clip to norm
and add Gaussian noise with variance
) to discriminator updates at each step of the training. Privacy of the generator is then guaranteed by the postprocessing property of BDP. It is worth mentioning, however, that clipping and/or adding noise to generator gradients can be beneficial for training in some cases, to keep a better balance in the game between the critic and the generator, and it should not be overlooked by developers.We choose not to implement more complicated schemes, such as PATEGAN [18] or GPATE [21], which use PATE framework [27]
to guarantee differential privacy for GANs. Our key rationale is that a more complicated structure of this solution could create unnecessary errors and additional privacy leakage (e.g. leaking privacy by backpropagating through the teachers’ votes to the generator, thereby neglecting the added noise). Nevertheless, we show in our evaluation that due to the distributioncalibrated BDP accounting (and hence, less added noise) our GAN generates better quality samples compared to these more complex solutions.
4.3 Privacy Analysis
In order to compute privacy guarantees of the synthetic dataset w.r.t. the real one, we need to bound privacy loss of the generative model. As noted before, we effectively enforce privacy on the critic and then rely on preservation of guarantees under postprocessing. This arrangement ensures a simple adoption of privacy accounting for discriminative models.
Privacy accounting is done by using the Bayesian accountant [32]
. To benefit from the data distribution information, it needs to sample a number of gradients at each iteration in addition to the one used in the update. These gradients are then used to estimate the upper confidence bound on the privacy cost
:(5) 
where
(6)  
(7) 
Here,
is the binomial distribution with
experiments (a hyperparameter) and the probability of success (equal to the probability of sampling a single data point in a batch), and are two gradient samples differing in one data point.The privacy guarantee is calculated from the privacy cost, by fixing either or :
(8) 
For more details on the Bayesian accountant and related proofs, see [32].
An important difference in privacy accounting for GANs is that not every update of the critic should be accounted for. Updates on fake data samples do not leak information about the real data beyond what is already accounted for in the previous iterations. Therefore, only real updates are sampled and used for the privacy cost estimation. In some GAN architectures, however, one should be careful to consider additional sources of privacy leakage, such as the gradient penalty in WGANGP [16].
To better understand how the BDP bound relates to the traditional DP, consider the following conditional probability:
(9) 
The moments accountant outputs that upperbounds for all . It is not true in general for other accounting methods, but let us focus on MA, as it is by far the most popular. Consequently, the moments accountant bound is
(10) 
where is a chosen constant. At the same time, BDP bounds the probability that is not conditioned on , but we can transform one to another through marginalisation and obtain:
(11) 
On the surface, this guarantee seems considerably weaker, as it holds only in expectation. However, since
is a nonnegative random variable in
, we can apply Markov’s inequality and obtain a tail bound on it using . We can therefore find a pair that holds for any percentile of the data/user distribution, not just in expectation. In all our experiments, we consider bounds well above 99th percentile, so it is very unlikely to encounter data for which the equivalent DP guarantee doesn’t hold.4.4 Federated Learning Case
In the above description, we did not make any assumptions on where the data are located. The most logical scenario to consider is federated learning, like in [5], such that the data remain on user devices at all times.
To accommodate FL scenarios, minimal modifications to the approach are required. Training of the generative model would be performed in the same way as any other federated model, and privacy accounting would be done at the userlevel [5]. Baysian DP results are also directly transferable to FL [33], and privacy bounds are generally even tighter in this case.
5 Evaluation
In this section, we describe the experimental setup, implementation, and evaluate our method on MNIST [20] and FashionMNIST [36] datasets.
5.1 Experimental Setting
Dataset  Nonprivate  Private classifier 
GPATE  Our approach 

MNIST  
FashionMNIST 
We evaluate two major applications of the technique. First, in Section 5.2, we show that the generated samples can be used for debugging ML model through data inspection, resembling tasks T1T4 from [5]. Second, we examine the quality of the downstream ML model trained directly on synthetic samples (Section 5.3), thus demonstrating a possibility of solving T5 (data labelling/annotation) as well.
In the debugging experiment, we attempt to detect a more subtle bug compared to [5]: an incorrect image rotation that yields lower model performance. While the pixel intensity inversion can be easily spotted using lowfidelity synthetic samples, image rotation requires higher fidelity to be detected.
Downstream learning experiments are set up as follows:

Train the generative model (teacher) on the original data under privacy guarantees.

Generate an artificial dataset by the obtained model and use it to train ML models (students).

Evaluate students on the heldout real test set.
We use two image datasets, MNIST and FashionMNIST. Both have training and test examples, where each example is a size greyscale image. The task of MNIST is handwritten digit recognition, while for FashionMNIST it is clothes type recognition. Although these datasets may not be of particular interest from the privacy viewpoint, this choice is defined by the ability to compare to prior work.
Our evaluation is implemented in Python and Pytorch
^{3}^{3}3http://pytorch.org. For the generative model, we experimented with variations of Wasserstein GAN [4] and WGANGP [16], but found the former to produce better results, probably because gradient clipping is already a part of the privacy mechanism. Our critic consists of three convolutional layers with SELU activations
[19] followed by a fully connected linear layer with another SELU and then a linear classifier. The generator starts with a fully connected linear layer that transforms noise (and possibly labels) into adimensional feature vector which is then passed through a SELU activation and three deconvolution layers with SELU activations. The output of the third deconvolution layer is downsampled by max pooling and normalised with a
tanhactivation function.5.2 Data Inspection
The data inspection experiment is setup in the following way. We introduce the rotation bug through randomly rotating some images by . We then train the two generative models, on correct images and on altered images, and compare their samples. We also train a model with DP to show that its image quality would not be sufficient to detect the error.
Figure 2 shows the output of generative models trained on MNIST with and without image rotation. By examining the samples, developers can clearly determine that a portion of images was rotated. This way, the error can be promptly identified and fixed. On the other hand, with generative models that uphold the traditional DP guarantee (Figure 1(c)), it would be difficult to detect such preprocessing error, because the produced samples have very low fidelity, even though in this case is unjustifiably high at the order of .
We also observe that the synthetic data quality under BDP (see Figures 1 and 1(a)) is sufficient to detect previously unseen classes or dataset biases, such as underrepresented classes. Moreover, these results are achieved with a strong privacy guarantee: under BDP, and hence, the probability that DP does not hold for this data is less than .
5.3 Learning Performance
Here, we evaluate the generalisation ability of the student model trained on artificial data. More specifically, we train a student model on generated data and report test classification accuracy on a real heldout set.
The goal of this experiment is to show that having a privacypreserving generative model we can use synthetic samples to fully replace the real data. Not only it allows to eliminate manual labelling of real (and potentially sensitive) data, but also expand the set of problems that can be solved by FL (task T5 in augenstein2019generative classification). For example, some medical data cannot be automatically annotated, and users are not qualified to do that, so highquality synthetic data would allow the annotation to be performed by doctors without privacy risks for users.
We imitate human annotation by training a separate classifier (with the same privacy guarantee as the generative model) and using it to label synthetic images. While this approach is somewhat different from prior work on generating data for training ML models, comparisons in this section are still valid because our annotator maintains the same privacy guarantee.
We choose to compare with the method called GPATE [21], because it is one of the best recent techniques in terms of privacyutility tradeoff. The authors showed that it outperforms another PATEbased approach, PATEGAN [18], as well as DPGAN [37], based on DPSGD.
Student model accuracy is shown in Table 1. Apart from GPATE, we compare our method to a nonprivate classifier trained directly on the real dataset, and a private classifier, trained on the real dataset with Bayesian DP. In the case of generative models, the same (nonprivate) classifier is trained on the private synthetic output. All results in the table are obtained with the privacy guarantee of DP, or BDP, which is equivalent to DP for this data with high probability. Although [21] report better results for DP, we do not include those in the study, because is too high for providing meaningful guarantee [32].
Generally, we observe that on these datasets switching from real to synthetic data does not significantly deteriorate accuracy of the student model while maintaining strong theoretical privacy guarantees. On MNIST, the drop in performance between a private discriminative and a private generative approach is less than . It is more noticeable on FashionMNIST, but is still below and is still lower than the drop between a nonprivate and a private classifiers. Moreover, as Figures 3 and 4 show, models trained on synthetic data achieve sufficiently good performance even when only a small portion of it is labelled. As little as 100 labelled samples is enough to outperform models trained on data generated with comparable DP guarantees.
Interestingly, nonprivate synthetic data (not shown in the table) allow to reach only marginally better results, suggesting that most of the accuracy loss comes from the generative model rather than privacy preservation. Figures 3 and 4 seem corroborate this finding, as synthetic data learning curve quickly saturates.
6 Conclusions
We explore the use of generative adversarial networks to tackle the problem of privacypreserving data inspection and annotation in machine learning. While the previous approaches to this problem involve generative models either without any privacy guarantee or with differential privacy, we opt for a different privacy notion – Bayesian differential privacy. By capturing the inherent properties of data and allowing for nonuniform privacy loss throughout the dataset, it enables higherfidelity synthetic data while still maintaining a privacy guarantee comparable to DP.
Our evaluation shows that privacypreserving GANs with BDP can be used to detect subtle bugs in data itself or preprocessing pipelines, which could not be caught by DP GANs due to low samples fidelity. Similarly, biases in the data and previously unseen classes can be discovered.
In addition, the generated data can be directly annotated and used for training in place of the real data. We demonstrate that student models trained on our synthetic samples achieve significantly higher accuracy compared to prior stateoftheart and exhibit only a mild drop in performance compared to private classification with real data. Furthermore, this gap is mainly determined by the quality of the generative model, and hence, will get smaller with advances in that field.
References
 [1] (2016) Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 308–318. Cited by: §1, §2, §2, §3.

[2]
(2013)
Differential privacy applications to bayesian and linear mixed model estimation
. Journal of Privacy and Confidentiality 5 (1), pp. 4. Cited by: §2.  [3] (1985) Exchangeability and related topics. In École d’Été de Probabilités de SaintFlour XIII—1983, pp. 1–198. Cited by: §3.
 [4] (2017) Wasserstein gan. arXiv preprint arXiv:1701.07875. Cited by: §5.1.
 [5] (2019) Generative models for effective ml on private, decentralized datasets. arXiv preprint arXiv:1911.06679. Cited by: §1, §4.4, §4.4, §5.1, §5.1.

[6]
(2017)
Privacypreserving generative deep neural networks support clinical data sharing
. bioRxiv, pp. 159756. Cited by: §2, §4.2.  [7] (2017) Practical secure aggregation for privacypreserving machine learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1175–1191. Cited by: §1.

[8]
(2018)
Composable and versatile privacy via truncated cdp.
In
Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing
, pp. 74–86. Cited by: §2.  [9] (2016) Concentrated differential privacy: simplifications, extensions, and lower bounds. In Theory of Cryptography Conference, pp. 635–658. Cited by: §2.
 [10] (2017) On the meaning and limits of empirical differential privacy. Journal of Privacy and Confidentiality 7 (3), pp. 3. Cited by: §2.
 [11] (2014) The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science 9 (3–4), pp. 211–407. Cited by: §3.
 [12] (2016) Concentrated differential privacy. arXiv preprint arXiv:1603.01887. Cited by: §2.
 [13] (200607) Differential privacy. In 33rd International Colloquium on Automata, Languages and Programming, part II (ICALP 2006), Vol. 4052, Venice, Italy, pp. 1–12. External Links: Link, ISBN 3540359079 Cited by: §1, §2.
 [14] (2015) Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 1322–1333. Cited by: §1, §2.
 [15] (2014) Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672–2680. Cited by: §1.
 [16] (2017) Improved training of wasserstein gans. In Advances in Neural Information Processing Systems, pp. 5769–5779. Cited by: §4.3, §5.1.
 [17] (2017) Deep models under the gan: information leakage from collaborative deep learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 603–618. Cited by: §1.
 [18] (2018) PATEgan: generating synthetic data with differential privacy guarantees. Cited by: §2, §4.2, §5.3.
 [19] (2017) Selfnormalizing neural networks. In Advances in Neural Information Processing Systems, pp. 972–981. Cited by: §5.1.
 [20] (1998) Gradientbased learning applied to document recognition. Proceedings of the IEEE 86 (11), pp. 2278–2324. Cited by: §5.
 [21] (2019) Scalable differentially private generative student model via pate. arXiv preprint arXiv:1906.09338. Cited by: §2, §4.2, §5.3, §5.3.
 [22] (2016) Communicationefficient learning of deep networks from decentralized data. arXiv preprint arXiv:1602.05629. Cited by: §1, §2.
 [23] (2012) Informationtheoretic foundations of differential privacy. In International Symposium on Foundations and Practice of Security, pp. 374–381. Cited by: §2.
 [24] (2009) Computational differential privacy. In Annual International Cryptology Conference, pp. 126–142. Cited by: §2.
 [25] (2017) Renyi differential privacy. In Computer Security Foundations Symposium (CSF), 2017 IEEE 30th, pp. 263–275. Cited by: §2.
 [26] (2016) Semisupervised knowledge transfer for deep learning from private training data. arXiv preprint arXiv:1610.05755. Cited by: §2.
 [27] (2018) Scalable private learning with pate. arXiv preprint arXiv:1802.08908. Cited by: §2, §4.2.
 [28] (2015) A new method for protecting interrelated time series with bayesian prior distributions and synthetic data. Journal of the Royal Statistical Society: Series A (Statistics in Society) 178 (4), pp. 963–975. Cited by: §2.
 [29] (2015) Privacypreserving deep learning. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pp. 1310–1321. Cited by: §2.
 [30] (2017) Membership inference attacks against machine learning models. In Security and Privacy (SP), 2017 IEEE Symposium on, pp. 3–18. Cited by: §1, §2.
 [31] (20190812) Federated generative privacy. In IJCAI Workshop on Federated Machine Learning for User Privacy and Data Confidentiality (FML 2019), External Links: Link Cited by: §2, §2, §5.1.
 [32] (2019) Bayesian differential privacy for machine learning. arXiv preprint arXiv:1901.09697. Cited by: §1, §2, §3, §4.3, §4.3, §5.3.
 [33] (2019) Federated learning with bayesian differential privacy. arXiv preprint arXiv:1911.10071. Cited by: §4.4, §5.1.
 [34] (2018) Towards demystifying membership inference attacks. arXiv preprint arXiv:1807.09173. Cited by: §1.
 [35] (2016) On the relation between identifiability, differential privacy, and mutualinformation privacy. IEEE Transactions on Information Theory 62 (9), pp. 5018–5029. Cited by: §2.
 [36] (20170828)(Website) External Links: cs.LG/1708.07747 Cited by: §5.
 [37] (2018) Differentially private generative adversarial network. arXiv preprint arXiv:1802.06739. Cited by: §2, §4.2, §5.3.
 [38] (2018) Differentially private releasing via deep generative model. arXiv preprint arXiv:1801.01594. Cited by: §2.
Comments
There are no comments yet.