With machine learning (ML) becoming ubiquitous in many aspects of our society, questions of its privacy and security take centre stage. A growing field of research in privacy attacks on ML [14, 30, 17, 34] tells us that it is possible to infer information about training data even in a black-box setting, without access to model parameters. A wider population, however, is concerned with privacy practices used in the ML development cycle, such as company employees or contractors manually inspecting and annotating user data111https://www.theguardian.com/technology/2020/jan/10/skype-audio-graded-by-workers-in-china-with-no-security-measures222https://www.bloomberg.com/news/articles/2019-04-10/is-anyone-listening-to-you-on-alexa-a-global-team-reviews-audio.
provides a rigorous theoretical guarantee, which states (informally) that the algorithm output would not significantly change when a single user adds or removes their data, except with small (failure) probability. Another approach gaining popularity isfederated learning (FL) [22, 7], where a central entity trains a model by computing updates locally on-device and then securely aggregating these updates on a server. This way user data never leave their devices.
In spite of significant progress, neither of these approaches solves the problem of manual data labelling. Moreover, it creates an additional hurdle for developers, as they cannot inspect data, especially in decentralised settings, making it difficult to understand the model behaviour and find bugs in data and implementations. augenstein2019generative augenstein2019generative provide a more complete characterisation of these questions.
This paper follows augenstein2019generative augenstein2019generative in adopting generative adversarial networks (GAN)  trained in a privacy-preserving manner for addressing these issues. More specifically, we use the notion of Bayesian differential privacy (BDP) , which takes into account the data distribution and provides a more meaningful guarantee for in-distribution samples than classical DP. Intuitively, when DP has uniform failure probability for all data points, BDP allows it to be non-uniform, thereby discounting points that are naturally difficult to hide and providing a strong guarantee for the rest of the dataset. Since both can use the same obfuscation mechanism, while computing two privacy bounds in parallel, a DP guarantee would still hold for out-of-distribution samples. More details on the overall approach and privacy are provided in Section 4.
The advantage of using this privacy definition is that it enables generating data of higher fidelity, compared to previous work on GANs with DP, allowing for finer-grained inspection of data. While some problems with data or data pipelines can be discovered using very coarse samples (e.g. pixel intensity inversion in ), more subtle bugs, like partial data corruption, would require samples of much better quality, rendering the DP guarantee too loose to be meaningful. Moreover, if fidelity is high enough, synthetic data can be used for annotation and training itself, removing the related privacy concerns and extending applicability of FL. We evaluate our solution in these two aspects in Section 5.
The main contributions of this paper are as follows:
we use Bayesian DP to enable higher quality GAN samples, while still providing a strong privacy guarantee;
we demonstrate that this technique can be used to discover finer data errors than has been previously reported;
we also show that for some tasks synthetic data are of high enough quality to be used for labelling and training.
2 Related Work
A rapidly expanding area of privacy-preserving machine learning research has been recently focused on the attacks that compromise privacy of training data, such as model inversion  and membership inference . The former is based on observing the output probabilities of the target model for a given class and performing gradient descent on an input reconstruction. The latter assumes an attacker with access to similar data, which is used to train ”shadow” models, mimicking the target, and the attack model, which predicts if a certain example has already been seen during training based on its output probabilities. Both attacks can be performed in a black-box setting, without access to the model internal parameters.
Differential privacy (DP)  is widely accepted as the gold standard for preventing such attacks. One of the early takes on the problem is to use disjoint datasets and distributed training with DP. For example,  propose to train a model in a distributed manner by communicating sanitised updates from participants to a central authority. Such a method, however, yields high privacy losses [1, 26]. An alternative technique suggested by  also uses disjoint training sets and builds an ensemble of independently trained teacher models to transfer knowledge to a student model by labelling public data. This result has been extended in  to achieve state-of-the-art image classification results in a private setting (with single-digit DP bounds). A different approach is taken by 
. They propose using differentially private stochastic gradient descent (DP-SGD) to train deep learning models in a private manner. This approach achieves high accuracy maintaining relatively low DP bounds and being simpler to implement, but may also require pre-training on public data.
Due to the fact that the DP threat model is extremely broad, achieving a reasonable guarantee may be difficult or even impossible. For this reason, a number of alternative definitions has been proposed over the recent years, aimed at relaxing the guarantee or providing tighter composition bounds under certain assumptions. Examples are computational DP , mutual-information privacy [23, 35], different versions of concentrated DP (CDP , zCDP , tCDP ), and RényiDP (RDP) . Some other relaxations [2, 28, 10] tip the balance even further in favour of applicability at the cost of weaker guarantees, for example considering the average-case instead of the worst-case .
In this work, we rely on another relaxation, called Bayesian differential privacy . This notion utilises the fact that data come from a particular distribution, and not all data samples are equally likely (e.g. unlikely to find a sound record among ECG samples). At the same time, it maintains a similar probabilistic interpretation of its parameters and
. It is worth noting, that unlike some of the relaxations mentioned above, Bayesian DP can provide a tail bound on privacy loss, similarly to the moments accountant (MA), and is not limited to a particular dataset, but rather a particular type of data (e.g. emails, MRI images, etc.), which is a much more permitting assumption.
Up until recently, another aspect of privacy in machine learning has been largely overlooked: the human involvement in the development cycle and manual data processing. These issues can be mitigated, at least partially, by federated learning (FL) , which brings a great promise for user privacy. Yet, FL paradigm creates additional problems of its own. augenstein2019generative augenstein2019generative provide a good starting point, systematising these problems and proposing a solution by the use of synthetic data. Although privacy-preserving data synthesis using GANs has been introduced in earlier works [6, 37, 38, 31, 18, 21], these papers mainly focused on achieving high utility of synthetic data without addressing a broader scope of privacy leakage via manual data handling.
A common problem of privacy-preserving GANs, however, is that the generated samples have very low fidelity, unless the privacy guarantee is unreasonably weak. Our approach makes progress in exactly this perspective: we can achieve much higher quality outputs with little compromise in privacy guarantees (and only for outliers that are difficult to hide). As a result, our synthetic data yield better performance of downstream analytics, and simultaneously, provide more powerful data inspection capabilities.
In this section, we provide some background useful for understanding the paper.
We use to represent neighbouring (adjacent) datasets. If not specified, it is assumed that these datasets differ in a single example. Individual examples in a dataset are denoted by or , while the example by which two datasets differ—by . We assume , whenever possible to do so without loss of generality. The private learning outcomes (i.e. noised gradients) are denoted by .
A randomised function (mechanism) with domain and range satisfies -differential privacy if for any two adjacent inputs and for any set of outcomes the following holds:
Privacy loss of a randomised mechanism for inputs and outcome takes the following form:
The Gaussian noise mechanism achieving -DP, for a function , is defined as
where and is the L2-sensitivity of .
A randomised function (algorithm) with domain and range satisfies -(weak) Bayesian differential privacy if for any two adjacent datasets , differing in a single data point , and for any set of outcomes the following holds:
While the definition of BDP is very close to that of DP, there are some important differences: the interpretation of is slightly different, data are assumed to come from a distribution (although it is not required to be known), and samples are assumed to be exchangeable . Nonetheless, this notion remains applicable in a wide range of practical scenarios .
In parts of the paper, we refer to augenstein2019generative classification of ML developer tasks, which can be condensed to:
T1 - Sanity checking data.
T2 - Debugging mistakes.
T3 - Debugging unknown labels / classes.
T4 - Debugging poor performance on certain classes / slices / users.
T5 - Human labelling of examples.
T6 - Detecting bias in the training data.
4 Our Approach
In this section, we describe our approach, intuition behind it, its privacy analysis, and discuss how to extend it to federated learning settings.
The primary distinction of Bayesian differential privacy is that it takes into account the data distribution, and by extension, assumes that all data points are drawn from the same distribution, although these distributions may be multimodal, highly complex, and generally unknown. This is a natural hypothesis in many machine learning applications, but especially so when working with generative models like GANs.
The task of generative modelling in itself is to learn an underlying data distribution, and thus, a common distribution is an implicit belief. This results in an organic match with BDP, because there are no assumptions to add to the problem.
Another part of our intuition is that the foremost source of privacy leakage are outliers. On the one hand, their respective privacy loss would be discounted in BDP accounting due to their low probability. On the other hand, we can reduce the number of samples generated by the GAN to decrease the chances of these outliers appearing in the synthetic dataset.
We are given a dataset of labelled () or unlabelled () examples. This dataset can be decentralised, in which case we would use FL (see Section 4.4). Our task is to train a GAN, which consists of the generator and the critic (discriminator), to generate synthetic samples from .
and add Gaussian noise with variance) to discriminator updates at each step of the training. Privacy of the generator is then guaranteed by the post-processing property of BDP. It is worth mentioning, however, that clipping and/or adding noise to generator gradients can be beneficial for training in some cases, to keep a better balance in the game between the critic and the generator, and it should not be overlooked by developers.
to guarantee differential privacy for GANs. Our key rationale is that a more complicated structure of this solution could create unnecessary errors and additional privacy leakage (e.g. leaking privacy by backpropagating through the teachers’ votes to the generator, thereby neglecting the added noise). Nevertheless, we show in our evaluation that due to the distribution-calibrated BDP accounting (and hence, less added noise) our GAN generates better quality samples compared to these more complex solutions.
4.3 Privacy Analysis
In order to compute privacy guarantees of the synthetic dataset w.r.t. the real one, we need to bound privacy loss of the generative model. As noted before, we effectively enforce privacy on the critic and then rely on preservation of guarantees under post-processing. This arrangement ensures a simple adoption of privacy accounting for discriminative models.
Privacy accounting is done by using the Bayesian accountant 
. To benefit from the data distribution information, it needs to sample a number of gradients at each iteration in addition to the one used in the update. These gradients are then used to estimate the upper confidence bound on the privacy cost:
is the binomial distribution withexperiments (a hyper-parameter) and the probability of success (equal to the probability of sampling a single data point in a batch), and are two gradient samples differing in one data point.
The privacy guarantee is calculated from the privacy cost, by fixing either or :
For more details on the Bayesian accountant and related proofs, see .
An important difference in privacy accounting for GANs is that not every update of the critic should be accounted for. Updates on fake data samples do not leak information about the real data beyond what is already accounted for in the previous iterations. Therefore, only real updates are sampled and used for the privacy cost estimation. In some GAN architectures, however, one should be careful to consider additional sources of privacy leakage, such as the gradient penalty in WGAN-GP .
To better understand how the BDP bound relates to the traditional DP, consider the following conditional probability:
The moments accountant outputs that upper-bounds for all . It is not true in general for other accounting methods, but let us focus on MA, as it is by far the most popular. Consequently, the moments accountant bound is
where is a chosen constant. At the same time, BDP bounds the probability that is not conditioned on , but we can transform one to another through marginalisation and obtain:
On the surface, this guarantee seems considerably weaker, as it holds only in expectation. However, since
is a non-negative random variable in, we can apply Markov’s inequality and obtain a tail bound on it using . We can therefore find a pair that holds for any percentile of the data/user distribution, not just in expectation. In all our experiments, we consider bounds well above 99th percentile, so it is very unlikely to encounter data for which the equivalent DP guarantee doesn’t hold.
4.4 Federated Learning Case
In the above description, we did not make any assumptions on where the data are located. The most logical scenario to consider is federated learning, like in , such that the data remain on user devices at all times.
To accommodate FL scenarios, minimal modifications to the approach are required. Training of the generative model would be performed in the same way as any other federated model, and privacy accounting would be done at the user-level . Baysian DP results are also directly transferable to FL , and privacy bounds are generally even tighter in this case.
5.1 Experimental Setting
We evaluate two major applications of the technique. First, in Section 5.2, we show that the generated samples can be used for debugging ML model through data inspection, resembling tasks T1-T4 from . Second, we examine the quality of the downstream ML model trained directly on synthetic samples (Section 5.3), thus demonstrating a possibility of solving T5 (data labelling/annotation) as well.
In the debugging experiment, we attempt to detect a more subtle bug compared to : an incorrect image rotation that yields lower model performance. While the pixel intensity inversion can be easily spotted using low-fidelity synthetic samples, image rotation requires higher fidelity to be detected.
Downstream learning experiments are set up as follows:
Train the generative model (teacher) on the original data under privacy guarantees.
Generate an artificial dataset by the obtained model and use it to train ML models (students).
Evaluate students on the held-out real test set.
We use two image datasets, MNIST and Fashion-MNIST. Both have training and test examples, where each example is a size greyscale image. The task of MNIST is handwritten digit recognition, while for Fashion-MNIST it is clothes type recognition. Although these datasets may not be of particular interest from the privacy viewpoint, this choice is defined by the ability to compare to prior work.
Our evaluation is implemented in Python and Pytorch333http://pytorch.org. For the generative model, we experimented with variations of Wasserstein GAN  and WGAN-GP 
, but found the former to produce better results, probably because gradient clipping is already a part of the privacy mechanism. Our critic consists of three convolutional layers with SELU activations followed by a fully connected linear layer with another SELU and then a linear classifier. The generator starts with a fully connected linear layer that transforms noise (and possibly labels) into a
-dimensional feature vector which is then passed through a SELU activation and three deconvolution layers with SELU activations. The output of the third deconvolution layer is down-sampled by max pooling and normalised with atanhactivation function.
5.2 Data Inspection
The data inspection experiment is setup in the following way. We introduce the rotation bug through randomly rotating some images by . We then train the two generative models, on correct images and on altered images, and compare their samples. We also train a model with DP to show that its image quality would not be sufficient to detect the error.
Figure 2 shows the output of generative models trained on MNIST with and without image rotation. By examining the samples, developers can clearly determine that a portion of images was rotated. This way, the error can be promptly identified and fixed. On the other hand, with generative models that uphold the traditional DP guarantee (Figure 1(c)), it would be difficult to detect such pre-processing error, because the produced samples have very low fidelity, even though in this case is unjustifiably high at the order of .
We also observe that the synthetic data quality under BDP (see Figures 1 and 1(a)) is sufficient to detect previously unseen classes or dataset biases, such as under-represented classes. Moreover, these results are achieved with a strong privacy guarantee: under -BDP, and hence, the probability that -DP does not hold for this data is less than .
5.3 Learning Performance
Here, we evaluate the generalisation ability of the student model trained on artificial data. More specifically, we train a student model on generated data and report test classification accuracy on a real held-out set.
The goal of this experiment is to show that having a privacy-preserving generative model we can use synthetic samples to fully replace the real data. Not only it allows to eliminate manual labelling of real (and potentially sensitive) data, but also expand the set of problems that can be solved by FL (task T5 in augenstein2019generative classification). For example, some medical data cannot be automatically annotated, and users are not qualified to do that, so high-quality synthetic data would allow the annotation to be performed by doctors without privacy risks for users.
We imitate human annotation by training a separate classifier (with the same privacy guarantee as the generative model) and using it to label synthetic images. While this approach is somewhat different from prior work on generating data for training ML models, comparisons in this section are still valid because our annotator maintains the same privacy guarantee.
We choose to compare with the method called G-PATE , because it is one of the best recent techniques in terms of privacy-utility trade-off. The authors showed that it outperforms another PATE-based approach, PATE-GAN , as well as DP-GAN , based on DP-SGD.
Student model accuracy is shown in Table 1. Apart from G-PATE, we compare our method to a non-private classifier trained directly on the real dataset, and a private classifier, trained on the real dataset with Bayesian DP. In the case of generative models, the same (non-private) classifier is trained on the private synthetic output. All results in the table are obtained with the privacy guarantee of -DP, or -BDP, which is equivalent to -DP for this data with high probability. Although  report better results for -DP, we do not include those in the study, because is too high for providing meaningful guarantee .
Generally, we observe that on these datasets switching from real to synthetic data does not significantly deteriorate accuracy of the student model while maintaining strong theoretical privacy guarantees. On MNIST, the drop in performance between a private discriminative and a private generative approach is less than . It is more noticeable on Fashion-MNIST, but is still below and is still lower than the drop between a non-private and a private classifiers. Moreover, as Figures 3 and 4 show, models trained on synthetic data achieve sufficiently good performance even when only a small portion of it is labelled. As little as 100 labelled samples is enough to outperform models trained on data generated with comparable DP guarantees.
Interestingly, non-private synthetic data (not shown in the table) allow to reach only marginally better results, suggesting that most of the accuracy loss comes from the generative model rather than privacy preservation. Figures 3 and 4 seem corroborate this finding, as synthetic data learning curve quickly saturates.
We explore the use of generative adversarial networks to tackle the problem of privacy-preserving data inspection and annotation in machine learning. While the previous approaches to this problem involve generative models either without any privacy guarantee or with differential privacy, we opt for a different privacy notion – Bayesian differential privacy. By capturing the inherent properties of data and allowing for non-uniform privacy loss throughout the dataset, it enables higher-fidelity synthetic data while still maintaining a privacy guarantee comparable to DP.
Our evaluation shows that privacy-preserving GANs with BDP can be used to detect subtle bugs in data itself or pre-processing pipelines, which could not be caught by DP GANs due to low samples fidelity. Similarly, biases in the data and previously unseen classes can be discovered.
In addition, the generated data can be directly annotated and used for training in place of the real data. We demonstrate that student models trained on our synthetic samples achieve significantly higher accuracy compared to prior state-of-the-art and exhibit only a mild drop in performance compared to private classification with real data. Furthermore, this gap is mainly determined by the quality of the generative model, and hence, will get smaller with advances in that field.
-  (2016) Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 308–318. Cited by: §1, §2, §2, §3.
Differential privacy applications to bayesian and linear mixed model estimation. Journal of Privacy and Confidentiality 5 (1), pp. 4. Cited by: §2.
-  (1985) Exchangeability and related topics. In École d’Été de Probabilités de Saint-Flour XIII—1983, pp. 1–198. Cited by: §3.
-  (2017) Wasserstein gan. arXiv preprint arXiv:1701.07875. Cited by: §5.1.
-  (2019) Generative models for effective ml on private, decentralized datasets. arXiv preprint arXiv:1911.06679. Cited by: §1, §4.4, §4.4, §5.1, §5.1.
Privacy-preserving generative deep neural networks support clinical data sharing. bioRxiv, pp. 159756. Cited by: §2, §4.2.
-  (2017) Practical secure aggregation for privacy-preserving machine learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1175–1191. Cited by: §1.
Composable and versatile privacy via truncated cdp.
Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, pp. 74–86. Cited by: §2.
-  (2016) Concentrated differential privacy: simplifications, extensions, and lower bounds. In Theory of Cryptography Conference, pp. 635–658. Cited by: §2.
-  (2017) On the meaning and limits of empirical differential privacy. Journal of Privacy and Confidentiality 7 (3), pp. 3. Cited by: §2.
-  (2014) The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science 9 (3–4), pp. 211–407. Cited by: §3.
-  (2016) Concentrated differential privacy. arXiv preprint arXiv:1603.01887. Cited by: §2.
-  (2006-07) Differential privacy. In 33rd International Colloquium on Automata, Languages and Programming, part II (ICALP 2006), Vol. 4052, Venice, Italy, pp. 1–12. External Links: Cited by: §1, §2.
-  (2015) Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 1322–1333. Cited by: §1, §2.
-  (2014) Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672–2680. Cited by: §1.
-  (2017) Improved training of wasserstein gans. In Advances in Neural Information Processing Systems, pp. 5769–5779. Cited by: §4.3, §5.1.
-  (2017) Deep models under the gan: information leakage from collaborative deep learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 603–618. Cited by: §1.
-  (2018) PATE-gan: generating synthetic data with differential privacy guarantees. Cited by: §2, §4.2, §5.3.
-  (2017) Self-normalizing neural networks. In Advances in Neural Information Processing Systems, pp. 972–981. Cited by: §5.1.
-  (1998) Gradient-based learning applied to document recognition. Proceedings of the IEEE 86 (11), pp. 2278–2324. Cited by: §5.
-  (2019) Scalable differentially private generative student model via pate. arXiv preprint arXiv:1906.09338. Cited by: §2, §4.2, §5.3, §5.3.
-  (2016) Communication-efficient learning of deep networks from decentralized data. arXiv preprint arXiv:1602.05629. Cited by: §1, §2.
-  (2012) Information-theoretic foundations of differential privacy. In International Symposium on Foundations and Practice of Security, pp. 374–381. Cited by: §2.
-  (2009) Computational differential privacy. In Annual International Cryptology Conference, pp. 126–142. Cited by: §2.
-  (2017) Renyi differential privacy. In Computer Security Foundations Symposium (CSF), 2017 IEEE 30th, pp. 263–275. Cited by: §2.
-  (2016) Semi-supervised knowledge transfer for deep learning from private training data. arXiv preprint arXiv:1610.05755. Cited by: §2.
-  (2018) Scalable private learning with pate. arXiv preprint arXiv:1802.08908. Cited by: §2, §4.2.
-  (2015) A new method for protecting interrelated time series with bayesian prior distributions and synthetic data. Journal of the Royal Statistical Society: Series A (Statistics in Society) 178 (4), pp. 963–975. Cited by: §2.
-  (2015) Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pp. 1310–1321. Cited by: §2.
-  (2017) Membership inference attacks against machine learning models. In Security and Privacy (SP), 2017 IEEE Symposium on, pp. 3–18. Cited by: §1, §2.
-  (2019-08-12) Federated generative privacy. In IJCAI Workshop on Federated Machine Learning for User Privacy and Data Confidentiality (FML 2019), External Links: Cited by: §2, §2, §5.1.
-  (2019) Bayesian differential privacy for machine learning. arXiv preprint arXiv:1901.09697. Cited by: §1, §2, §3, §4.3, §4.3, §5.3.
-  (2019) Federated learning with bayesian differential privacy. arXiv preprint arXiv:1911.10071. Cited by: §4.4, §5.1.
-  (2018) Towards demystifying membership inference attacks. arXiv preprint arXiv:1807.09173. Cited by: §1.
-  (2016) On the relation between identifiability, differential privacy, and mutual-information privacy. IEEE Transactions on Information Theory 62 (9), pp. 5018–5029. Cited by: §2.
-  (2017-08-28)(Website) External Links: Cited by: §5.
-  (2018) Differentially private generative adversarial network. arXiv preprint arXiv:1802.06739. Cited by: §2, §4.2, §5.3.
-  (2018) Differentially private releasing via deep generative model. arXiv preprint arXiv:1801.01594. Cited by: §2.