Exploring the Space of Adversarial Images
Adversarial examples have raised questions regarding the robustness and security of deep neural networks. In this work we formalize the problem of adversarial images given a pretrained classifier, showing that even in the linear case the resulting optimization problem is nonconvex. We generate adversarial images using shallow and deep classifiers on the MNIST and ImageNet datasets. We probe the pixel space of adversarial images using noise of varying intensity and distribution. We bring novel visualizations that showcase the phenomenon and its high variability. We show that adversarial images appear in large regions in the pixel space, but that, for the same task, a shallow classifier seems more robust to adversarial images than a deep convolutional network.READ FULL TEXT VIEW PDF
Multiple different approaches of generating adversarial examples have be...
Deep neural networks (DNNs) have been demonstrated to be vulnerable to
Recently, many adversarial examples have emerged for Deep Neural Network...
Deep Neural Networks (DNNs) are known to be susceptible to adversarial
We present a novel method for generating robust adversarial image exampl...
Recent studies have demonstrated the vulnerability of deep convolutional...
We propose a retrieval-augmented convolutional network and propose to tr...
Exploring the Space of Adversarial Images
Small but purposeful pixel distortions can easily fool the best deep convolutional networks for image classification [1, 2]. The small distortions are hardly visible by humans, but still can mislead most neural networks. Adversarial images can even fool the internal representation of images by neural networks 
. That problem has divided the Machine Learning community, with some hailing it as a “deep flaw” of deep neural networks; and others promoting a more cautious interpretation, and showing, for example, that most classifiers are susceptible to adversarial examples [5, 6].
or using evolutionary algorithms.
Despite the controversy, adversarial images surely suggest a lack of robustness, since they are (for humans) essentially equal to correctly classified images. Immunizing a network against those perturbations increases its ability to generalize, a form of regularization 
whose statistical nature deserves further investigation. Even the traditional backpropagation training procedure can be improved with adversarial gradient. Recent work shows that it is possible for the training procedure to make classifiers robust to adversarial examples by using a strong adversary 9].
In this paper, we extend previous works on adversarial images for deep neural networks , by exploring the pixel space of such images using random perturbations. That framework (Figure 1) allows us to ask interesting questions about adversarial images. Initial skepticism about the relevance of adversarial images suggested they existed as isolated points in the pixel space, reachable only by a guided procedure with complete access to the model. More recent works [5, 10] claim that they inhabit large and contiguous regions in the space. The correct answer has practical implications: if adversarial images are isolated or inhabit very thin pockets, they deserve much less worry than if they form large, compact regions. In this work we intend to shed light to the issue with an in-depth analysis of adversarial image space.
Assume we have a pre-trained classifier that, for each inputof the image belonging to the class label . We can assign to the label corresponding to the highest probability . Assume further that , for grayscale images, or for RGB images, where and are the lower and upper limits of the pixel scale.
Assume that is the correct label and that we start with , otherwise there is no point in fooling the classifier. We want to add the smallest distortion to , such that the highest probability will no longer be assigned to . The distortions must keep the input inside its space, i.e., we must ensure that . In other words, the input is box-constrained. Thus, we have the following optimization:
That formulation is more general than the one presented by , for it ignores non-essential details, such as the choice of the adversarial label. It also showcases the non-convexity: since is convex, the inequality is clearly concave , making the problem non-trivial even if the model were linear in
. Deep networks, of course, exacerbate the non-convexity due to their highly non-linear model. For example, a simple multi-layer perceptron for binary classification could have, which is neither convex nor concave due to the hyperbolic tangent.
Training a classifier usually means minimizing the classification error by changing the model weights. To generate adversarial images, however, we hold the weights fixed, and find the minimal distortion that still fools the network.
We can simplify the optimization problem of eq. 1 by exchanging the
inequality for a term in the loss function that measures how adversarial the probability output is:
where we introduce the adversarial probability target , which assigns zero probability to all but a chosen adversarial label . That formulation is essentially the same of , picking an explicit (but arbitrary) adversary label. We stipulate the loss function: the cross-entropy () between the probability assignments; while  keep that choice open.
The constant balances the importance of the two objectives. The lower the constant, the more we will minimize the distortion norm. Values too low, however, may turn the optimization unfeasible. We want the lowest, but still feasible, value for .
We can solve the new formulation with any local search compatible with box-constraints. Since the optimization variables are the pixel distortions, the problem size is exactly the size of the network input, which in our case varies from for MNIST  to for OverFeat . In contrast to current neural network training, that reaches hundreds of millions of weights, those sizes are small enough to allow second-order procedures, which converge faster and with better guarantees . We chose L-BFGS-B, a box-constrained version of the popular L-BFGS second-order optimizer . We set the number of corrections in the limited-memory matrix to 15, and the maximum number of iterations to 150. We used Torch7 to model the networks and extract their gradient with respect to the inputs . Finally, we implemented a bisection search to determine the optimal value for . The algorithm is explained in detail in the next section.
Algorithm 1 implements the optimization procedure used to find the adversarial images. The algorithm is essentially a bisection search for the constant , where in each step we solve a problem equivalent to Eq. 2. Bisection requires initial lower and upper bounds for , such that the upper bound succeeds in finding an adversarial image, and the lower bound fails. It will then search the transition point from failure to success (the “zero” in a root-finding sense): that will be the best . We can use as lower bound, as it always leads to failure (the distortion will go to zero). To find an upper bound leading to success, we start from a very low value, and exponentially increase it until we succeed. During the search for the optimal we use warm-starting in L-BFGS-B to speed up convergence: the previous optimal value found for is used as initial value for the next attempt.
, with hundreds of classes, this search would be too costly. Instead, in our experiments, we opt to consider the adversarial label as one of the sources of random variability. As we will show, this does not upset the analyses.
The source code for adversarial image generation and pixel space analysis can be found in https://github.com/tabacof/adversarial.
In this section we explore the vector space spanned by the pixels of the images to investigate the “geometry” of adversarial images: are they isolated, or do they exist in dense, compact regions? Most researchers currently believe that images of a certain appearance (and even meaning) are contained into relatively low-dimensional manifolds inside the whole space . However, those manifolds are exceedingly convoluted, discouraging direct geometric approaches to investigate the pixel space.
Thus, our approach is indirect, probing the space around the images with small random perturbations. In regions where the manifold is nice — round, compact, occupying most of the space — the classifier will be consistent (even if wrong). In the regions where the manifold is problematic — sparse, discontinuous, occupying small fluctuating subspaces — the classifier will be erratic.
To allow comparison with the results of , we employ the MNIST handwritten digits database (10 classes, 60k training and 10k testing images), and the 2012 ImageNet Large Visual Recognition Challenge Dataset (1000 classes, 1.2M+ training and 150k testing images).
For MNIST, 
tested convolutional networks and autoencoders. We employ both convolutional networks and a logistic linear classifier. While logistic classifiers have limited accuracy (7.5% error), their training procedure is convex . They also allowed us to complement the original results  by investigating adversarial images in a shallow classifier.
The convolutional network we employed for MNIST/ConvNet consisted of two convolutional layers with 64 and 128 55 filters, two 2
2 max-pooling layers after the convolutional layers, one fully-connected layer with 256 units, and a softmax layer as output. We used ReLU for nonlinearity, and 0.5 dropout before the two last layers. The network was trained with SGD and momentum. Without data augmentation, this model achieves 0.8% error on the test set.
For ImageNet, we used the pre-trained OverFeat network , which achieved 4th place at the ImageNet competition in 2013, with 14.2% top-5 error in the classification task, and won the localization competition the same year.  employed AlexNet , which achieved 1st place at the ImageNet competition in 2012, with 15.3% top-5 error.
On each dataset, we preprocess the inputs by standardizing each pixel with the global mean and standard deviation of all pixels in the training set images.
Figure 2 illustrates all three cases. Original and adversarial images are virtually indistinguishable. The pixel differences (middle row) do not show any obvious form — although a faint “erasing-and-rewriting” effect can be observed for MNIST. Figures (a)a and (b)b also suggest that the MNIST classifiers are more robust to adversarial images, since the distortions are larger and more visible. We will see, throughout the paper, that classifiers for MNIST and for ImageNet have important differences in how they react to adversarial images.
Each case (MNIST/Logistic, MNIST/ConvNet, ImageNet/OverFeat) was investigated independently, by applying the optimization procedure explained in Section II-A. For ImageNet we sampled 5 classes (Abaya, Ambulance, Banana, Kit Fox, and Volcano), 5 correctly classified examples from each class, and sampled 5 adversarial labels (Schooner, Bolete, Hook, Lemur, Safe), totaling 125 adversarial images. For MNIST, we just sampled 125 correctly classified examples from the 10K examples in the test set, and sampled an adversarial label (from 9 possibilities) for each one. All random sampling was made with uniform probability. To sample only correctly classified examples, we rejected the misclassified ones until we accumulated the needed amount. We call, in the following sections, those correctly classified images originals, since the adversarial images are created from them.
The probing procedure consisted in picking an image pair (an adversarial image and its original), adding varying levels of noise to their pixels, resubmitting both to the classifier, and observing if the newly assigned labels corresponded to the original class, to the adversarial class, or to some other class.
We measured the levels of noise () relative to the difference between each image pair. We initially tested a Gaussian i.i.d. model for the noise. For each image , our procedure creates an image where , and and
are the sample mean and variance of the distortion pixels. In the experiments we rangedfrom to . To keep the pixel values of within the original range we employ . In practice, we observed that clamping has little effect on the noise statistics.
An i.i.d. Gaussian model discards two important attributes of the distortions: spatial correlations, and higher-order momenta. We wanted to evaluate the relative importance of those, and thus performed an extra round of experiments that, while still discarding all spatial correlations by keeping the noise i.i.d., adds higher momenta information by modeling non-parametrically the distribution of distortion pixels. Indeed, a study of those higher momenta (Table I) suggests that the adversarial distortions has a much heavier tail than the Gaussians, and we wanted to investigate how that affects the probing. The procedure is exactly the same as before, but with , where is an empirical distribution induced by a non-parametric observation of the distortion pixels. In those experiments we cannot control the level: the variance of the noise is essentially the same as the variance of the distortion pixels.
Our main metric is the fraction of images (in %) that keep or switch labels when noise is added to a departing image, which we use as a measure of the stability of the classifier at the departing image in the pixel space. The fraction is computed over a sample of 100 probes, each probe being a repetition of the experiment with all factors held fixed but the sampling of the random noise.
Figure 3 shows that adversarial images do not appear isolated. On the contrary, to completely escape the adversarial pocket we need to add a noise with much higher variance — notice that the horizontal axis is logarithmic — than the distortion used to reach the adversarial image in the first place.
In both networks, the original images display a remarkable robustness against Gaussian noise (Figures (b)b and (f)f), confirming that robustness to random noise does not imply robustness to adversarial examples . That shows that while the adversarial pockets are not exactly isolated, neither are they as well-behaved as the zones that contain the correctly classified samples.
The results in Figure 3 are strongly averaged, each data point summarizing, for a given level of noise, the result of 125 experiments: the fraction of images that fall in each label for all five original class labels, all five original samples from each label, and all five adversarial class labels. In reality there is a lot of variability that can be better appreciated in Figure 4. Here each curve alongside the axis experiments represents a single choice of original class label, original sample, and adversarial class label, thus there are 125 curves. (The order of the curves along this axis is arbitrary and chosen to minimize occlusions and make the visualization easier). The graphs show that depending on a specific configuration, the label may be very stable and hard to switch (curves that fall later or do not fall at all), or very unstable (curves that fall early). Those 3D graphs also reinforce the point about the stability of the correctly classified original images.
The results suggest that the classifiers for MNIST are more resilient against adversarial images than ImageNet/OverFeat. Moreover, the shallow MNIST/logistic behaves differently than the deep MNIST/ConvNet, as shown by the the “falling columns” in Figure 3: initially, a small push in MNIST/logistic throws a larger fraction of the adversarial examples back to the correct space. However, at large noise levels, MNIST/logistc saturates with a larger fraction of images still adversarial than MNIST/ConvNet.
Finally, we wanted to investigate how the nature of the noise added affected the experiments. Recall that our i.i.d. Gaussian noise differs from the original optimized distortion in two important aspects: no spatial correlations, and no important higher-order momenta. To explore the influence of those two aspects, we introduced a noise modeled after the empirical distribution of the distortion pixels. This still ignores spatial correlations, but captures higher-order momenta. The statistics of the distortion pixels are summarized in Table I, and reveal a distribution that is considerably heavier-tailed than the Gaussians we have employed so far.
Figure 5 contrasts the effect of this noise modeled non-parametrically after the distortions with the effect of the comparable Gaussian noise (). Each point in the curves is one of the 125 experiments, and represents the fraction of the 100 probe images that stays in the same class as the departing — adversarial or original — image. The experiments where ordered by this value in each curve (thus the order of the experiments in the curves is not necessarily the same). Here the individual experiments are not important, but the shape of the curves: how early and how quickly they fall.
For ImageNet, the curves for the non-parametric noise (dotted lines) fall before the curves for the Gaussian noise (continuous line), showing that, indeed, the heavier tailed noise affects the images more, even without the spatial correlation. In addition, all curves fall rather sharply. This shows that in almost all experiments, either all probes stay in the same label as the original, or all of them switch. Few experiments present intermediate results. This rather bimodal behavior was already present in the curves of Figure 4.
For MNIST, again, the effect is different: Gaussian and heavy-tail noises behave much more similarly and the curves fall much more smoothly.
Our in-depth analysis reinforces previous claims found in the literature [5, 10]: adversarial images are not necessarily isolated, spurious points: many of them inhabit relatively dense regions of the pixel space. This helps to explain why adversarial images tend to stay adversarial across classifiers of different architectures, or trained on different sets: slightly different classification boundaries stay confounded by the dense adversarial regions.
The nature of the noise affects the resilience of both adversarial and original images. The effect is clear in ImageNet/OverFeat, where a Gaussian noise affects the images less than a heavy-tailed noise modeled after the empirical distribution of the distortions used to reach the adversarial images in the first place. An important next step in the exploration, in our view, is to understand the spatial nature of the adversarial distortions, i.e., the role spatial correlations play.
Recent works have attributed susceptibility to adversarial attacks to the linearity in the network , but our experiments suggest the phenomenon may be more complex. A weak, shallow, and relatively more linear classifier (logistic regression), seems no more susceptible to adversarial images than a strong, deep classifier (deep convolutional network), for the same task (MNIST). A strong deep model on a complex task seems to be more susceptible. Are adversarial images an inevitable Achilles’ heel of powerful complex classifiers? Speculative analogies with the illusions of the Human Visual System are tempting, but the most honest answer is that we still know too little. Our hope is that this article will keep the conversation about adversarial images ongoing and help further explore those intriguing properties.
We would like to thank Micael Carvalho for his helpful hints and revision; Jonghoon Jin for open access to his OverFeat wrapper111https://github.com/jhjin/overfeat-torch; and Soumith Chintala, and the rest of the Torch7 team for all of their help on the mailing list. We thank the Brazilian agencies CAPES, CNPq and FAPESP for financial support.
R. Bi, “Does deep learning have deep flaws?”http://www.kdnuggets.com/2014/06/deep-learning-deep-flaws.html, 2014, accessed: 2015-09-08.
Y. LeCun, C. Cortes, and C. J. Burges, “The mnist database of handwritten digits,” 1998.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” inAdvances in neural information processing systems, 2012, pp. 1097–1105.