1 Introduction
Adversarial examples, inputs to machine learning models that an adversary designs to manipulate model output, pose a major concern in machine learning applications. Many hypotheses have been suggested in the literature trying to explain the existence of adversarial examples. For example,
Tanay and Griffin (2016) hypothesise that these examples lie near the decision boundary, while Nguyen et al. (2015) hypothesise that these examples lie in low density regions of the input space. However, adversarial examples can lie far from the decision boundary (e.g. “garbage” images (Nguyen et al., 2015)), and using a simple spheres dataset it was shown that adversarial examples can exist in high density regions as well (Gilmer et al., 2018). In parallel work following Nguyen et al. (2015)’s lowdensity hypothesis, Li (2018) empirically modelled input image density on MNIST and successfully detected adversarial examples by thresholding low input density. This puzzling observation, seemingly inconsistent with the spheres experiment in (Gilmer et al., 2018), suggests that perhaps additional conditions beyond possessing the input density have led to the observed robustness by Li (2018).Suggesting two sufficient conditions, here we prove that an idealised model (in a sense defined below) cannot have adversarial examples, neither in low density nor in high density regions of the input space. We concentrate on adversarial examples in discriminative classification models, models which are used in practical applications. To formalise our treatment, and to gain intuition into the results, we use tools such as discriminative Bayesian neural network (BNN) classifiers (MacKay, 1992; Neal, 1995)
together with their connections to modern techniques in deep learning such as stochastic regularisation techniques
(Gal, 2016). This pragmatic Bayesian perspective allows us to shed some new light on the phenomenon of adversarial examples. We further discuss which models other than BNNs abide by our conditions. Our hypothesis suggests why MC dropoutbased techniques are sensible for adversarial examples identification, and why these have been observed to be consistently effective against a variety of attacks (Li and Gal, 2017; Feinman et al., 2017; Rawat et al., 2017; Carlini and Wagner, 2017).We support our hypothesis mathematically and experimentally using HMC and dropout inference. We construct a synthetic dataset derived from MNIST for which we can calculate ground truth input densities, and use this dataset to demonstrate that model uncertainty correlates to input density, and that under our conditions this density is low for adversarial examples. Using our newfound insights we develop a new attack for MC dropoutbased models which does not require gradient information, by looking for “holes” in the epistemic uncertainty estimation, i.e. imperfections in the uncertainty approximation, and suggest a mitigation technique as well. We give illustrative examples using MNIST (LeCun and Cortes, 1998), and experiment with realworld catsvsdogs image classification tasks (Elson et al., 2007) using a VGG13 variant (Simonyan and Zisserman, 2015).
2 Related Literature
There has been much discussion in the literature about the nature of “adversarial examples”. Introduced in Szegedy et al. (2013) using gradient crafting techniques for image inputs, these were initially hypothesised to be similar to the rational numbers, a dense set within the set of all images^{1}^{1}1This was refuted in (Goodfellow et al., 2014); Below we will see another simple theoretical argument refuting this hypothesis.. Szegedy et al. (2013)’s gradient based crafting method performed a targeted attack, where an input image is perturbed with a small perturbation to classify differently to the original image class. Followup research by Goodfellow et al. (2014) introduced nontargeted
attacks, where a given input image is perturbed to an arbitrary wrong class by following the gradient away from the image label. This crafting technique also gave rise to a new type of adversarial examples, “garbage” images, which look nothing like the original training examples yet classify with high output probability.
Goodfellow et al. (2014) showed that the deep neural networks’ (NNs) nonlinearity property is not the cause of vulnerability to adversarial examples, by demonstrating the existence of adversarial examples in linear models as well. They hypothesised that NNs are very linear by design and that in highdimension spaces this is sufficient to cause adversarial examples. Later work studied the linearity hypothesis further by constructing linear classifiers which do not suffer from the phenomenon (Tanay and Griffin, 2016). Instead, Tanay and Griffin (2016) argued that adversarial examples exist when the classification boundary lies close to the manifold of sampled data.After the introduction of adversarial examples by Szegedy et al. (2013), Nguyen et al. (2015)
developed crafting techniques which do not rely on gradients but rather use genetic algorithms to generate “garbage” adversarial examples.
Nguyen et al. (2015) further hypothesised that such adversarial examples have low probability under the data distribution, and that joint density models will be more ‘robust’ because the low marginal probability would be indicative of an example being adversarial. Nguyen et al. (2015)argued that this mitigation is not practical though since current generative models do not scale well to complex highdimensional data distributions such as ImageNet.
Li (2018)recently extended these ideas to nongarbage adversarial examples as well, and lent support to the hypothesis by showing on MNIST that a deep naive Bayes classifier (a generative model) is able to detect targeted adversarial examples by thresholding low input density. Parallel work to
Li (2018) has also looked at the hypothesis of adversarial examples having to exist in low input density regions, but proposed that adversarial examples can exist in high density regions as well. More specifically, Gilmer et al. (2018)construct a simple dataset composed of a uniform distribution over two concentric spheres in high dimensions, with a deterministic feedforward NN trained on 50M random samples from the two spheres. They propose an attack named “manifold attack” which constrains the perturbed adversarial examples to lie on one of the two concentric spheres, i.e. in a region of high density, and demonstrate that the attack successfully finds adversarial examples with a model trained on the spheres dataset. This demonstration that there
could exist adversarial examples on the data manifold and in high input density regions falsifies the hypothesis that adversarial examples must exist in low density regions of the input space, and is seemingly contradictory to the evidence presented in Li (2018). We will resolve this inconsistency below.A parallel line to the above research has tried to construct bounds on the minimum magnitude of the perturbation required for an image to become adversarial. Fawzi et al. (2018) for example quantify “robustness” using an introduced metric of expected perturbation magnitude and derive an upper bound on a model’s robustness. Fawzi et al. (2018)’s derivation relies on some strong assumptions, for example assuming that it is feasible to compute the distance between an input and the set for some classifier . Papernot et al. (2016) further give a definition of a robust model, extending the definitions of Fawzi et al. (2018) to targeted attacks, and propose a model to satisfy this definition. In more recent work, Peck et al. (2017) extend on both these ideas ((Fawzi et al., 2018), (Papernot et al., 2016)), and propose a lower bound on the robustness to perturbations necessary to change the classification of a neural network. Peck et al. (2017) also make strong assumptions in their premise, assuming the existence of an oracle able to assign a “correct” label for each input . This assumption is rather problematic since it implies that any input has a “correct” class, including completely blank images which have no objects in them. Lastly, Hein and Andriushchenko (2017), working in parallel to (Peck et al., 2017), use alternative assumptions and instead offer a bound relying on local Lipschitz continuity.
Following the perturbation bounds literature, in this work we will use similar but simpler tools, relying on the continuity of the classifier alone. Contrary to the generative modelling perspective, we will concentrate on discriminative Bayesian models which are much easier to scale to highdimensional data (Kendall and Gal, 2017). Such models capture information about the density of the training set as we will see below. We will define our idealised models under some strong assumptions (as expected from an idealised model), in a similar fashion to previous research concerned with provable guarantees. However below we will also give empirical support demonstrating the ideas we develop with practical tools. The class of models which satisfy our conditions postulated below includes models other than BNNs, such as RBF networks and nearest neighbour in feature space. Even so, we will formalise our arguments in ‘BNN terminology’ to keep precise and rigorous language. After laying out our ideas, below we will discuss which other models our results extend to as well.
3 Background
A deep neural network for classification is a function from an input space (e.g. images) to a set of labels (e.g. ). The network is parametrised by a set of weights and biases , which are generally chosen to minimize some empirical risk
on the model outputs and the target outputs over some dataset
with and. Rather than thinking of the weights as fixed parameters to be optimized over, the Bayesian approach is to treat them as random variables, and so we place a prior distribution
over the weights of the network. If we also have a likelihood function that gives the probability of given a set of parameter values and an input to the network, then we can conduct Bayesian inference given a dataset by marginalising (integrating out) the parameters. Such models are known as Bayesian neural networks (MacKay, 1992; Neal, 1995). The conditional probability of the model parameters given a training set is known as the posterior distribution. Ideally we would integrate out our uncertainty by taking the expectation of the predictions over the posterior, rather than using a point estimate of the parameters (e.g. MAP, the maximiser of the posterior). For deep Bayesian neural networks this marginalisation cannot be done analytically. Several approximate inference techniques exist, and here we will concentrate on two of them. Hamiltonian Monte Carlo (HMC) (Neal, 1995) is considered to be the ‘goldstandard’ in inference, but does not scale well to large amounts of data. It has been demonstrated to give stateoftheart results on many smallscale tasks involving uncertainty estimation in nontractable models (Neal, 1995). A more pragmatic alternative is approximate variational inference, e.g. with dropout approximating distributions (Gal, 2016). This technique is known to scale to large models, preserving model accuracy, while giving useful uncertainty estimates for various downstream tasks (Kendall and Gal, 2017). However, dropout approximate inference is known to give worse calibrated approximating distributions, a fact we highlight below as well.Bayesian neural networks are tightly connected to Gaussian processes (Rasmussen and Williams, 2006), and in fact the latter Gaussian processes can be seen as the infinite limit of single hidden layer Bayesian neural networks with Gaussian priors over their weights (Neal, 1995). Both can quantify “epistemic uncertainty”: uncertainty due to our lack of knowledge. In terms of machine learning, this corresponds to a situation where our model output is poorly determined due to lack of data near the input we are attempting to predict an output for. This is distinguished from “aleatoric uncertainty” (which we will refer to below as ambiguity) which is due to genuine stochasticity in the data (Kendall and Gal, 2017): This corresponds to noisy data, for example digit images that can be interpreted as either 1 or 7; no matter how much data the model has seen, if there is inherent noise in the labels then the best prediction possible may be a high entropy one (for example, if we train a model to predict fair coin flips, the best prediction is the maxentropy distribution ).
An attractive measure of uncertainty able to distinguish epistemic from aleatoric examples is the information gain between the model parameters and the data. Recall that the mutual information (MI) between two random variables (r.v.s) and is given by
with the entropy of r.v. . In terms of machine learning, the amount of information we would gain about the model parameters r.v. if we were to receive a label realisation for the r.v. for a new input , given the dataset , is then given by the difference between the predictive entropy and the expected entropy :
(1) 
Being uncertain about an input point implies that if we knew the label at that point we would gain information. Conversely, if the function output at an input is already well determined, then we would gain little information from obtaining the label. Thus, the MI is a measurement of the model’s epistemic uncertainty (in contrast to the predictive entropy which is high when either the epistemic uncertainty is high or when there is ambiguity, e.g. refer to the example of a fair coin toss). Note that the MI is always bounded between 0 and the predictive entropy.
To gain intuition into the different types of uncertainty in BNNs we shall look at BNN realisations in function space with a toy dataset. Our BNN defines a distribution over NN parameters, which induces a distribution over functions from the input space to the output space. Drawing multiple function realisations we see (Fig. 1) that all functions map the training set inputs to the outputs, but each function takes a different, rather arbitrary, value on points not in the train set. Assessing the discrepancy of these functions on a given input allows us to identify if the tested point is near the training data or not. In classification, having high enough discrepancy between the presoftmax functions’ values for a fixed input leads to lower output probability when averaged over the postsoftmax values. Thus any input far enough from the training set will have low output probability.
Illustration of function realisations in softmax space (left), in logit space (presoftmax, middle), as well as epistemic (orange, right) and aleatoric uncertainty (blue, right). Note the high epistemic uncertainty (
) in regions of the input space where many function explanations exist, and how predictive probability mean (dark blue, left panel) is close to uniform in these areas. Also note aleatoric uncertainty spiking in regions of ambiguity (transition from class 0 to class 1, depicted in the left panel).4 Preliminaries and Intuition
We start by informally discussing the sufficient conditions for idealised models—informally, models with zero training loss—to be robust to adversarial examples. We will give some simple examples to depict the intuition behind these conditions. In the next section we will formalise the conditions with a rigorous presentation and prove that under these conditions a model cannot have adversarial examples.
We need two key idealised properties to hold in order for a model not to have adversarial examples: idealised architecture (i.e. the model is invariant to all transformations the data distribution is invariant to), and ability to indicate when an input lies far from the valid input points (e.g. uncertainty is higher than some , or the nearest neighbour is further than some , in either case indicating ‘don’t know’ by giving a low confidence prediction). The first property ensures the model has high coverage, i.e. generalises well to all inputs the data distribution defines as ‘similar’ to train points. The second property ensures the model can identify points which are far from all previously observed points (and any transformations of the points that the data distribution would regard as the same). Together, given a nondegenerate train set sampled from the data distribution, these two properties allow us to define an idealised model that would accepts and classify correctly all points one would define as a valid inputs to the model, and reject all other points.
The core idea of our proof is that a continuous classification model output doesn’t change much within small enough neighbourhoods of points ‘similar’ to the training set points, at least not enough to change the training points’ predictions by more than some . A main challenge in carrying out a nonvacuous proof is to guarantee that such models generalise well, i.e. have high coverage. This is a crucial property, since many models are ‘trivially’ robust to adversarial examples by simply rejecting anything which is not identical to a previously observed training point. To carry out our proof we therefore implicitly augment the train set using all transformations extracted from the model and to which the model is invariant (and by the first condition, to which the data generating distribution is invariant). These transformations are implicitly extracted from the model architecture itself: For example, a translation invariant model will yield a train set augmented with translations. Thus the augmented train set might be infinite. We stress though that we don’t change the train set for the model training phase; the augmented train set is only used to carry out the proof. In practice one builds the transformations the data distribution is invariant to into the model.
The implicitly augmented training set is used to avoid the degeneracy of the model predicting well on the train set but not generalising to unseen points. To gain more intuition into the role and construction of the set of transformations , recall the spheres dataset from (Gilmer et al., 2018), built of two concentric spheres each labelled with a different class. If it were possible to train a model perfectly with all sphere points, then the model could not have adversarial examples on the sphere because each point on the sphere must be classified with the correct sphere label. However it is impossible to define a loss over an infinite training set in practice, and a practical alternative to training the model with infinite training points is to build the invariances we have in the data distribution into our model. In the case of the spheres dataset we build a rotation invariance into the model. Since our model is now rotation invariant it is enough to have a single training point from each sphere in order for the model to generalise to the entire data distribution, therefore a model trained with only two data points will generalise well (have high coverage). A rotation invariant model trained with the two points is thus identical to an idealised model trained with the infinite number of points on the sphere. Formalising these ideas with the spheres example, in our proof below we rely on the implicitly constructed set of rotations ; In the proof our train set (the two points) is augmented with the set of all rotations, thus yielding a set containing all points from the two spheres—in effect implicitly constructing an idealised model.
We next formalise the ideas above. Although the language we use next is rooted in BNNs, we will generalise these results to other idealised models in the following section.
5 Theoretical Justification
We now show that idealised discriminative Bayesian neural networks, capturing perfect epistemic uncertainty and data invariances, cannot have adversarial examples. Here we follow (Nguyen et al., 2015; Papernot et al., 2016) where an adversarial example is defined as follows.
Definition 1.
An adversarial example is a model input which either

lies far from the training data but is still classified with high output probability (e.g. ‘garbage’ images), or

an example which is formed of an input which classifies with high output probability, and a small perturbation , s.t. a prediction on is also made with high output probability, and the predicted class on differs from the predicted class on . The perturbation can be either perceptible or imperceptible.
Note that other types of adversarial examples exist in the literature (Yang et al., 2017; Grosse et al., 2017; Kreuk et al., 2018).
We start by setting our premise. We will develop our proof for a binary classification setting with continuous models (i.e. the model is discriminative and its output is a single probability between 0 and 1, continuous with the input ), with a finite training set sampled from some data distribution. Our first assumption is that the training data has no ambiguity:
Assumption 1.
There exist no which is labelled with both class 0 and class 1.
This requirement of lack of ambiguity will be clarified below. We define an threshold for a prediction to be said to have been made ‘with high output probability’: is defined as predicting class 1 with high output probability, and respectively is said to predict class 0 with high output probability (e.g. is a nice choice).
Our first definition is the set containing all transformations that our data is invariant under, e.g. might be a set containing translations and local deformations for image data:
Definition 2.
Let be the data distribution were i.i.d. sampled from. Define to be the set of all transformations s.t. for all .
Note that cannot introduce ambiguity into our training set. For brevity in the proof we overload and use it to denote the augmented training set , i.e. we augment with all the possible transformations on it (note that may now be infinite); We further augment and overload correspondingly so each is matched with the label corresponding to . Note that to guarantee full coverage (i.e. all input points with high probability for some under the data distribution must have high output probability under the model) one would demand , i.e. every point in the input space must belong to some trajectory generated by some point from the train set, or equivalently, all equivalence classes defined by must be represented in the train set. We next formalise what we mean by ‘idealised NN’:
Definition 3.
We define an ‘idealised NN’ to be a NN which outputs probability 1 for each training set point with label 1, and outputs probability 0 on training set points with label 0. We further define a ‘Bayesian idealised NN’ to be a Bayesian model average of idealised NNs (i.e. we place a distribution over idealised NNs’ weights).
Note that this definition implies that the NN architecture is invariant to , our first condition for a model to be robust to adversarial examples. Model output (the predictive probability) for a Bayesian idealised NN is given by Bayesian model averaging: , which we write as for brevity. Note that a Bayesian idealised NN must have predictive probabilities taking one of the two values in on the training set.
Following Neal (1995) we know that infinitely wide (single hidden layer) BNNs converge to Gaussian processes (GPs) Rasmussen and Williams (2006). In more recent results, Matthews et al. showed that even finite width BNNs with more than a single hidden layer share many properties with GPs. Of particular interest to us is the GP’s epistemic uncertainty property (uncertainty which can increase ‘far’ from the training data, where far is defined using the GP’s lengthscale parameter)^{2}^{2}2Note that this property depends on the GP’s kernel; we discuss this in the next section.. We next formalise what we mean by ‘epistemic uncertainty’.
Definition 4.
We define ‘epistemic uncertainty’ to be the mutual information between the model parameters r.v. and the model output r.v. .
Denoting the model output probability by , we abuse notation slightly and write instead of for our Bernoulli r.v. with mean . Note that the mutual information satisfies . Since we assumed there exists no ambiguous in the dataset , we have for all .
Next we introduce a supporting lemma which we will use in our definition of an ‘idealised BNN’:
Lemma 1.
Let be the model output of some Bayesian idealised NN on input with training set . There exists for each such that the model predicts with high output probability on all in the deltaball^{3}^{3}3A delta ball around is defined as . .

Let be a training point. By Bayesian idealised NN definition, takes a value from . W.l.o.g. assume . By continuity of the BNN’s output there exists a s.t. all in the delta ball have model output probability larger than . Similarly for , there exists a s.t. all in the delta ball have model output probability smaller than . I.e. the model output probability is as that of up to an , and the model predicts with high output probability within the deltaball. ∎
Finally, we define an ‘idealised BNN’ to be a Bayesian idealised NN which has a ‘GP like’ distribution over the function space (where the GP’s kernel should account for the invariances which are built into the BNN model architecture, see for example (van der Wilk et al., 2017)), and which increases its uncertainty ‘fast enough’. Or more formally:
Definition 5.
We define an idealised BNN to be a Bayesian idealised NN with epistemic uncertainty higher than outside , the union of balls surrounding the training set points.
This is our second condition which must be satisfied for a model to be robust to adversarial examples.
We now have the tools required to state our main result:
Theorem 1.
Under the assumptions and definitions above, an idealised Bayesian neural network cannot have adversarial examples.

Let . By lemma 1, every perturbation that is under the delta ball does not change class prediction. Further, by the idealised BNN definition and epistemic uncertainty definition, we have that for all outside , with the model output probability on denoted as , the entropy satisfies . By symmetry, entropy of being larger than the entropy of means that , i.e. the prediction is with low output probability for both class 0 and class 1.
We have that every has either 1) , in which case , i.e. is classified with low output probability and cannot be adversarial, or 2) , in which case is within some delta ball with centre and label or . In the former case, i.e. is classified correctly with high output probability, and in the latter case , and is still classified correctly with high output probability. Since every perturbed input that is under a delta ball does not change the predicted class from that of the training example , cannot be adversarial either. ∎
Note that the assumption of lack of dataset ambiguity in the proof above can be relaxed, and the proof easily generalises to datasets with more than two classes. Next we look at the proof critically, followed by an assessment of the ideas developed above empirically, approximating the idealised BNN with HMC sampling.
5.1 Proof critique
We start by clarifying why we need to assume no ambiguity in the dataset. Simply put, if we had two pairs and for some in our dataset, then no NN can be idealised following our definition (i.e. give probability 1 to the first observed point and probability 0 to the second). More generally, we want to avoid issues of low predictive probability near the training data; this assumption can be relaxed assuming aleatoric noise and adapting the proof to use the mutual information rather than the entropy.
We use the idealised model architecture condition (and the set of transformations ) to guarantee good coverage in our proof. CNNs (or capsules, etc.) capture the invariances we believe we have in our data generating distribution, which is the ‘maxim’ representation learning uses to generalise well. Note though that it might very well be that the model that we use in practice is not invariant to all transformations we would expect the data generating distribution to be invariant to. That would be a failure case leading to limited coverage; Compare to the spheres dataset example – if our model can’t capture the rotation invariances then it might unjustifiably “reject” test points (i.e. classify them with low output probability, thus reduce coverage). In practice it is very difficult to define what transformations the data distribution is invariant to with realworld data distributions. However, we can estimate model coverage (to guarantee that the model generalises better than a lookup table or nearest neighbours) by empirical means as well. For example, we observe empirically on a variety of realworld tasks that CNNs have low uncertainty on test images which were sampled from the same data distribution as the train images, as we see in our experiments below and in other works (Kendall and Gal, 2017). In fact, there is a connection between a model’s generalisation error and its invariance to transformations to which the data distribution is invariant, which we discuss further in appendix A. This suggests that existing models with real data do capture sensible invariances from the dataset, enough to be regarded empirically as generalising well.
Next we look at the proof above in a critical way. First, note that our argument does not claim the existence of an idealised BNN. Ours is not an ‘existence’ proof. Rather, we proved that under the definition above of an idealised BNN, such a BNN cannot have adversarial examples. The interesting question which follows is ‘do there exist realworld BNNs and inference which approximately satisfy the definition?’. We attempt to answer this question empirically in the next section. Further note that our idealised BNN definition cannot hold for all possible BNN architectures. For a BNN to approximate our definition it has to increase its uncertainty fast enough. Empirically, for many practical BNN architectures the uncertainty indeed increases far from the data (Gal, 2016)
. For example, a single hidden layer BNN with sine activation functions converges to a GP with an RBF kernel as the number of BNN units increases
(Gal and Turner, 2015); Both the RBF GP and the finite BNN possess the desired property of uncertainty increasing far from the training set (Gal and Turner, 2015). This property has also been observed to hold empirically for deep ReLU BNNs
Gal (2016). In the same way that our results depend on the model architecture, not all GPs will be robust to adversarial examples either (e.g. a GP could increase uncertainty too slowly or not at all); This depends on the choice of kernel and kernel hyperparameters. The requirement for the uncertainty to increase quickly enough within a region where the function does not change too quickly raises interesting questions about the relation between Lipschitz continuity and model uncertainty. We hypothesise that a relation could be established between the Lipschitz constant of the BNN and its uncertainty estimates.Finally, our main claim in this work is that the idealised Bayesian equivalents of some of these other practical NN architectures will not have adversarial examples; In the experiments section below we demonstrate that realistic BNN architectures (e.g. deep ReLU models for MNIST classification), with nearidealised inference, approximate the property of perfect uncertainty defined above, and further show that practical approximate inference such as dropout inference approximates some of the properties but fails for others.
5.2 Adversarial examples on the spheres dataset
Gilmer et al. (2018) construct adversarial examples by constraining the perturbed example to lie on one of the spheres, i.e. in high input density regions. Gilmer et al. (2018) then demonstrate that it is possible to have adversarial examples lying in both high density and low density regions of the input space. Here we refined this argument and showed that adversarial examples must exist only in low density regions of the input space when the model captures relevant data invariances; I.e. when the model is built to capture the data invariances then adversarial examples must lie in low density regions of the input space and cannot exist in high density regions.
Further, an idealised BNN which is rotation invariant will increase its uncertainty for offmanifold adversarial examples (since it never saw them before), and the onmanifold examples will be part of the implicit set induced by the invariances and the model will thus classify them correctly. Therefore, the idealised BNN with the rotation invariance will have seen the true labels for all points on the manifold, and being idealised and able to classify such points correctly it will not have adversarial examples, neither on the manifold, nor off the manifold.
5.3 Generalisation to other idealised models
Our proof trivially generalises to other idealised models that satisfy the two conditions set above (idealised architecture and idealised ability to indicate invalid inputs – definition 5 for the case of idealised BNN models). In appendix B we discuss which idealised models other than BNNs satisfy these two conditions, and further justify why we chose to continue our developments below empirically studying nearidealised BNNs.
6 Empirical Evidence
In this section we give empirical evidence supporting the arguments above. We demonstrate the ideas using nearperfect epistemic uncertainty obtained from HMC (considered ‘gold standard’ for inference with BNNs (Neal, 1995)), and with image data for which we know the groundtruth imagespace density. We show that image density diminishes as images become adversarial, that uncertainty correlates with image density, and that stateoftheart adversarial crafting techniques fail with HMC. We then test how these ideas transfer to nonidealised data and models, demonstrating failures of dropout uncertainty on MNIST, and propose a new attack and a mitigation to this attack. We finish by assessing the robustness of our mitigation with a VGG13 variant.
6.1 Idealised case
In this subsection we are only concerned with ‘near idealised’ data and inference, assessing the definitions in the previous section. We start by deriving a new image dataset from MNIST (LeCun and Cortes, 1998), for which we know the ground truth density in the image space for each example , and are therefore able to determine how far away it is from the data distribution.
Our dataset, Manifold MNIST (MMNIST) was constructed as follows. We first trained a variational autoencoder (VAE) on MNIST with a 2D latent space. We chose three image classes (0, 1, and 4), discarding the latents of all other classes, and put a small ‘Gaussian bump’ on each latent point from our 3 classes. Summing the bumps for each latent class we get an analytical density corresponding to this class. We then discarded the MNIST latents, and defined the mixture of the 3 analytical densities in latent space as our ground truth image density (each mixture component identified with its corresponding ground truth class). Generating 5,000 samples from this mixture and decoding each sample using our fixed VAE decoder, we obtained our training set for which each image has a ground truth density (Fig. 3, see appendix C for density calculation). Note that this dataset does not satisfy our lackofdataambiguity assumption above, as seen in the figure.
First we show that the density decreases on average for image as we make adversarial (adding perturbations) using a standard LeNet NN
classifier as implemented in Keras
(LeCun et al., 1998; Chollet, 2015). Multiple images were sampled from our synthetic dataset, with the probability of an image in the input space plotted as it becomes adversarial for both targeted and nontargeted FGM (Goodfellow et al., 2014) attacks (Fig. 3). Together with Fig. 3, trajectories from the targeted attack (FGM) on MMNIST, seen in Fig. 13(a) in appendix D, show that even when the adversarial images still resemble the original images, they already have low probability under the dataset. Further, Fig. 13(b) shows that the deterministic NN accuracy on these images has fallen, i.e. the generated images successfully fool the model.Next, we show that nearperfect epistemic uncertainty correlates to density under the image manifold. We use given by a grid of equally spaced poitns over the 2D latent space (Fig. 6). We used a BNN with LeNet architecture and HMC inference to estimate the epistemic uncertainty (Fig. 6, visualised in the VAE latent space; Shown in white is uncertainty, calculated by decoding each latent point into image space, and evaluating the MI between the decoded image and the model parameters; A lighter background corresponds to higher uncertainty). In Fig. 6 we show that uncertainty correlates to density on the images from .
Finally, we show that adversarial crafting fails for HMC. In this experiment we sample a new realisation from the HMC predictive distribution with every gradient calculation, in effect approximating the infinite ensemble defined by an idealised BNN. We used a nontargeted attack (MIM, first place in the NIPS 2017 competition for adversarial attacks (Dong et al., 2017)), which was shown to fool finite deterministic ensembles and be robust to gradient noise. Table 1 shows success rate in changing test image labels for HMC and a deterministic NN, for maximum allowed input perturbation of sizes^{4}^{4}4Note that here we use to denote maximum perturbation magnitude, as is common in the literature, not to be confused with our from the proof. , v.s. a control experiment of simply adding noise of magnitude . Also shown average image entropy. Note HMC BNN success rate for the attack is similar to that of the noise, v.s. Deterministic where random noise does not change prediction much, but a structured perturbation fools the model very quickly. Note further that HMC BNN’s entropy increases quickly, showing that the model has many different possible output values for the perturbed images.
HMC BNN  Deterministic NN  

Adv. succ.  Noise succ.  Adv.  Noise  Adv. succ.  Noise succ.  Adv.  Noise  
6.2 Nonidealised case
Here we compare realworld inference (specifically, dropout) to nearperfect inference (HMC) on real noisy data (MNIST). We use the same encoder as in the previous section to visualise the model’s epistemic uncertainty in 2D (Fig. 11). Note the dropout uncertainty ‘holes’ compared to HMC. We plot the dropout MI v.s. HMC MI for the grid of points as before in Fig. 11.
6.3 New attack and defence
We use the dropout failure case above to suggest a new attack generating ‘garbage’ images with high output probability, which does not require gradient information but instead queries the model for its confidence: First, collect a dataset of images, and project to 2D. Gridup the latent space (Fig. 17 in appendix D) and query the model for uncertainty on each grid point. Order by distance from the nearest training point, and decode the farthest latents with low MI (i.e. points far from the training set on which the model is confident). Example crafted images given in Fig. 11. We further suggest a mitigation here, using intuition from above: we use an ensemble of randomly initialised dropout models (Fig. 11), and show that ensemble correlation with HMC MI fixes the uncertainty ‘holes’ to a certain extent (Fig. 11). In the appendix (D) we give quantitative results comparing the success rate of the new attack to FGM’s success rate, and show that dropout ensemble is more robust to the stateoftheart MIM attack compared to a single dropout model. We further show that the equivalent Deterministic model ensemble uncertainty contains more uncertainty ‘holes’ than the dropout ensemble.
6.4 Realworld cats vs dogs classification
We extend the results above and show that an ensemble of dropout models is more robust than a single dropout model using a VGG13 (Simonyan and Zisserman, 2015) variant on the ASIRRA (Elson et al., 2007) cats and dogs classification dataset. We retrained a VGG13 variant ((Simonyan and Zisserman, 2015), with a reduced number of FC units) on the ASIRRA (Elson et al., 2007) cats and dogs classification dataset, with Concrete dropout (Gal et al., 2017) layers added before every convolution. We compared the robustness of a single Concrete dropout model to that of an ensemble following the experiment setup of (Smith and Gal, 2018). Here we used the FGM attack with and infinity norm. Example adversarial images are shown in Fig. 13. Table 2 shows the AUC of different MI thresholds for declaring ‘this is an adversarial example!’, for all images, as well as for successfully perturbed images only (S). Full ROC plots are given in Fig. 13. We note that the more powerful attacks succeed in fooling this VGG13 model, whereas dropout Resnet50 based models seem to be more robust (Smith and Gal, 2018). We leave the study of model architecture effect on uncertainty and robustness for future research.
Model  AUC  AUC (S) 

Concrete Dropout  
Concrete Dropout Ensemble 
7 Discussion
We presented several idealised models which could satisfy our set conditions for robustness, opening the door for research into how various practical tools can approximate our set conditions. We highlighted that the main difficulty with modern BNNs is not coverage, but rather that approximate inference doesn’t increase the uncertainty fast enough with practical BNN tools (we show this in figures 6(a), demonstrating that we have holes in the dropout uncertainty). In contrast, HMC (which is not scalable for practical applications) does not have such uncertainty holes. One of our main conclusions is therefore that we need improved inference techniques in BNNs.
Further, designing density models over complex data such as images is challenging, and the claim that we can extract this information from a probabilistic discriminative model is not straightforward. This result also gives intuition into why dropout, a technique shown to relate to Bayesian modelling, seems to be effective in identifying adversarial examples. Lastly, our analysis has practical implications for the field as well. It highlights questions of interest to direct future research, such as which model architectures satisfy the conditions above best, and reveals flaws with current approaches.
Acknowledgements
We thank Mark van der Wilk, Yingzhen Li, Ian Goodfellow, Nicolas Papernot, and others, for feedback and comments on this work. This research was supported by The Alan Turing Institute. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.
References
 Carlini and Wagner [2017] Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. arXiv preprint arXiv:1705.07263, 2017.
 Chollet [2015] François Chollet. Keras, 2015. URL https://github.com/fchollet/keras. GitHub repository.
 Dong et al. [2017] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Xiaolin Hu, Jianguo Li, and Jun Zhu. Boosting adversarial attacks with momentum. arXiv preprint arXiv:1710.06081, 2017.
 Elson et al. [2007] Jeremy Elson, John (JD) Douceur, Jon Howell, and Jared Saul. Asirra: A captcha that exploits interestaligned manual image categorization. In Proceedings of 14th ACM Conference on Computer and Communications Security (CCS). Association for Computing Machinery, Inc., October 2007.
 Fawzi et al. [2018] Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Analysis of classifiers’ robustness to adversarial perturbations. Machine Learning, 107(3):481–508, 2018.
 Feinman et al. [2017] Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint arXiv:1703.00410, 2017.
 Gal [2016] Yarin Gal. Uncertainty in deep learning. PhD thesis, University of Cambridge, 2016.
 Gal and Turner [2015] Yarin Gal and Richard Turner. Improving the Gaussian process sparse spectrum approximation by representing uncertainty in frequency inputs. In Proceedings of the 32nd International Conference on Machine Learning (ICML15), 2015.
 Gal et al. [2017] Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. arXiv preprint arXiv:1705.07832, 2017.
 Gilmer et al. [2018] Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S. Schoenholz, Maithra Raghu, Martin Wattenberg, and Ian J. Goodfellow. Adversarial spheres. CoRR, abs/1801.02774, 2018.
 Goodfellow et al. [2014] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
 Grosse et al. [2017] Kathrin Grosse, Nicolas Papernot, Praveen Manoharan, Michael Backes, and Patrick McDaniel. Adversarial examples for malware detection. In European Symposium on Research in Computer Security, pages 62–79. Springer, 2017.
 Hein and Andriushchenko [2017] Matthias Hein and Maksym Andriushchenko. Formal guarantees on the robustness of a classifier against adversarial manipulation. In Advances in Neural Information Processing Systems, pages 2263–2273, 2017.

Kendall and Gal [2017]
Alex Kendall and Yarin Gal.
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
In Advances in Neural Information Processing Systems 30 (NIPS), 2017.  Kreuk et al. [2018] Felix Kreuk, Assi Barak, Shir AvivReuven, Moran Baruch, Benny Pinkas, and Joseph Keshet. Adversarial examples on discrete sequences for beating wholebinary malware detection. CoRR, abs/1802.04528, 2018.

LeCun and Cortes [1998]
Yann LeCun and Corinna Cortes.
The MNIST database of handwritten digits, 1998.
 LeCun et al. [1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradientbased learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
 Li [2018] Yingzhen Li. Are generative classifiers more robust to adversarial attacks? ICLR workshop track, 2018.
 Li and Gal [2017] Yingzhen Li and Yarin Gal. Dropout inference in Bayesian neural networks with alphadivergences. arXiv preprint arXiv:1703.02914, 2017.

MacKay [1992]
David JC MacKay.
A practical Bayesian framework for backpropagation networks.
Neural Computation, 4(3):448–472, 1992.  [21] AGDG Matthews, J Hron, M Rowland, RE Turner, and Z Ghahramani. Gaussian process behaviour in wide deep neural networks. In Proceedings of the 6th International Conference on Learning Representations.
 Neal [1995] Radford M Neal. Bayesian learning for neural networks. PhD thesis, University of Toronto, 1995.

Nguyen et al. [2015]
Anh Nguyen, Jason Yosinski, and Jeff Clune.
Deep neural networks are easily fooled: High confidence predictions
for unrecognizable images.
In
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
, pages 427–436, 2015.  Papernot and McDaniel [2018] Nicolas Papernot and Patrick McDaniel. Deep knearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv:1803.04765, 2018.
 Papernot et al. [2016] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In Security and Privacy (SP), 2016 IEEE Symposium on, pages 582–597. IEEE, 2016.
 Peck et al. [2017] Jonathan Peck, Joris Roels, Bart Goossens, and Yvan Saeys. Lower bounds on the robustness to adversarial perturbations. In Advances in Neural Information Processing Systems, pages 804–813, 2017.
 Rasmussen and Williams [2006] Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2006. ISBN 026218253X.
 Rawat et al. [2017] Ambrish Rawat, Martin Wistuba, and MariaIrina Nicolae. Adversarial phenomenon in the eyes of Bayesian deep learning. arXiv preprint arXiv:1711.08244, 2017.
 Simonyan and Zisserman [2015] K. Simonyan and A. Zisserman. Very deep convolutional networks for largescale image recognition. In International Conference on Learning Representations, 2015.
 Smith and Gal [2018] Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detection. UAI, 2018.
 Szegedy et al. [2013] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
 Tanay and Griffin [2016] Thomas Tanay and Lewis Griffin. A boundary tilting persepective on the phenomenon of adversarial examples. arXiv preprint arXiv:1608.07690, 2016.
 van der Wilk et al. [2017] Mark van der Wilk, Carl Edward Rasmussen, and James Hensman. Convolutional Gaussian processes. In Advances in Neural Information Processing Systems, pages 2845–2854, 2017.
 Yang et al. [2017] Wei Yang, Deguang Kong, Tao Xie, and Carl A Gunter. Malware detection in adversarial settings: Exploiting feature evolutions and confusions in android apps. In Proceedings of the 33rd Annual Computer Security Applications Conference, pages 288–302. ACM, 2017.
Appendix
Appendix A Coverage
To see why low test error implies high coverage, we present a simple argument that relies on the idealised case of zero expected error (error w.r.t. the data distribution) and the assumption that test error is representative of the expected error. Define a model to be invariant to a transformation almost everywhere (a.e.) when for all up to a zero measure set (i.e. for almost all ); Assume that the data distribution has no ambiguity (i.e. a point with nonzero probability for must have zero probability for ). If the model has zero expected error then a.e. in with a corresponding having nonzero probability conditioned on , . For all transformations to which the data distribution is invariant, i.e. , there exists that has nonzero probability conditioned on as well, and therefore (from the lack of ambiguity assumption) it must hold that . Therefore the model is invariant to a.e. as well.
Empirically, we observe nearidealised HMC LeNet BNN and dropout LeNet variants to have low test error with test points following the same distribution as the train points (we got >99% accuracy on the MNIST test set with a dropout BNN in our experiments). We use this as evidence towards the claim that our suggested models generalise well (have high coverage). Further, zero coverage for invalid inputs (i.e. the model saying ‘don’t know’ for outofdistribution examples, for example by giving uniform probabilities) is a desired property of our model.
Note that to get full coverage (i.e. not rejecting a single valid point) we must assume that for every valid input there exists some transformation mapping some training point to . This is more difficult to formalise for nonidealised models though.
Appendix B Generalisation to other idealised models
Here we discuss which idealised models satisfy our two conditions. The class of idealised models which satisfy our defined properties above is varied. The question of interest is what models are most suitable for which task, and which models approximate the idealised properties best.
We contrast several idealised models on the spheres dataset [Gilmer et al., 2018] to assess which could and could not satisfy our conditions. We use the spheres dataset here since we know the set of transformations an idealised model must be invariant to (i.e. if a model can satisfy the first condition). We will look at a NN with ReLU nonlinearities (i.e. with no special invariances), a BNN with the same structure, a NN which is rotation invariant, a BNN which is rotation invariant, an RBF network (with either architecture), standard nearest neighbours, and nearest neighbours in feature space with some deterministic feature extractor, all using finite training sets.

A NN with no invariances can have adversarial examples (as demonstrated in [Gilmer et al., 2018]).

A NN with rotation invariances will not have adversarial examples on the spheres (following our argument in section 4). However, the NN might have ‘garbage’ adversarial examples which classify with high output probability far away from the data, where the model might be wrongly confident (we show this in our experiments below as well). An idealised NN can predict with arbitrary high output probability far away from the training set, and there is no way to enforce the model not to do so – we can’t iterate over all points ‘not in the train set’ and force a standard deep NN model to predict with near uniform output probability on these points.

A BNN with no invariances cannot have adversarial examples on the sphere but will have low coverage. It will output a prediction for the finite train set points, and will output ‘don’t know’ (i.e. nearuniform probability) for all other points on the sphere which it didn’t have in the train set. The BNN will not have garbage adversarial examples far from the data (since the model averages many different functions, each giving different values far from the data, in effect increasing its uncertainty and pushing the predictive probability to uniform).

A BNN with rotation invariances will have no adversarial examples on the sphere (following the arguments in points 2 and 3) and with full coverage for all sphere points. It will have no garbage adversarial examples (following the argument in point 3). As mentioned we still assume a finite training set for the BNN, but having the model rotation invariant makes this equivalent to the case in point 3 with an infinite train set which includes all sphere points.

An idealised RBF network which collapses to uniform prediction fast enough follows the same intuition of idealised BNNs above – both for a rotation invariant RBF network as well as for a model with no invariances.

Nearest neighbour which uses thresholding to declare ‘don’t know’ and with a finite dataset (again, all above also used finite dataset and invariances built into the model itself) will have no adversarial examples, but will have low coverage following the same arguments in point 3. The only way to fix the issue of low coverage with standard nearest neighbours is to explicitly assume that the model is trained with an infinite training set with all sphere points (in which case it will have proper coverage). Note the difference to the BNN / NN models which use a finite training set with invariances built into the model to implicitly augment the train set. A possible way to alleviate this issue is to perform nearest neighbour in feature space, building invariances into the distances nearest neighbours uses, allowing a finite training set to be used to get full coverage. This idea is developed further in [Papernot and McDaniel, 2018].
Probability thresholding using the Bayesian approach plays an important role in our proof, but is not the only way to declare ‘don’t know’ as we saw above. Note though that nearest neighbour thresholding is not trivial: Even though one might define for example ‘distance in input space to nearest neighbour’ in order to declare an output as ‘don’t know’, in practice thresholding the distance in input space (or, for that matter, in feature space) can affect points differently in different parts of the input space (e.g. we want to have low threshold in high density regions v.s. high threshold in low density regions). The Bayesian approach gives tools to do the thresholding in the output probability space, further allowing us to define a tolerance to false positives if our uncertainty is calibrated. Lastly, we note that we can’t simply define a third class (class 2) to indicate ‘don’t know’, even in the rotation invariant NN. The quotient group of all distinct points in our spheres dataset (after identifying all points on the surface of a sphere with some radius as identical to each other) is still infinite: It is all the nonnegative reals (corresponding to sphere radii). Out of these, one point (r=1) corresponds to class 1, one point (r=1.3) corresponds to class 0. In this case there exist infinitely many points that will be assigned class 2. The point of the augmented train set trick from the proof is to induce finite train sets over which we can define the invariant model loss (which is feasible in practice). But with the ‘don’t know’ class the train set (in either case) is infinite, which means it is infeasible to define a loss over it. For these reasons we chose to continue our developments studying idealised and nearidealised BNNs.
Appendix C Image density calculation
For our MMNIST dataset we have an analytical expression for the density in the latent space for each class : a Gaussian (with the latent variable). With this density we can calculate the density of an observed image by MC integration over the latent space:
with and . In practice we use importance sampling for density calculations. In Fig. 17 we show that the ground truth latent space density correlates strongly with the image density obtained from this estimator on test MMNIST images.
Appendix D More empirical results
We next assess the success rate of getting garbage images which classify with high output probability (), comparing our new latent space attack which does not use gradient information to the untargeted FGS attack (which does use gradients), on a dropout NN with MNIST. Results are shown in Table 3. Note though that the sample size used here is rather small (15 generated images).
Attack  Success rate 

Latent space attack  
Untargeted FGM 
Table 4 shows the robustness of dropout ensemble v.s. dropout with a MIM attack on MNIST; Note the improved robustness for the ensemble.
Perturbation magnitude  Dropout  Dropout ensemble 
