Exploring the Space of Adversarial Images

10/19/2015 ∙ by Pedro Tabacof, et al. ∙ University of Campinas 0

Adversarial examples have raised questions regarding the robustness and security of deep neural networks. In this work we formalize the problem of adversarial images given a pretrained classifier, showing that even in the linear case the resulting optimization problem is nonconvex. We generate adversarial images using shallow and deep classifiers on the MNIST and ImageNet datasets. We probe the pixel space of adversarial images using noise of varying intensity and distribution. We bring novel visualizations that showcase the phenomenon and its high variability. We show that adversarial images appear in large regions in the pixel space, but that, for the same task, a shallow classifier seems more robust to adversarial images than a deep convolutional network.



There are no comments yet.


page 1

page 3

page 5

page 6

Code Repositories


Exploring the Space of Adversarial Images

view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Small but purposeful pixel distortions can easily fool the best deep convolutional networks for image classification [1, 2]. The small distortions are hardly visible by humans, but still can mislead most neural networks. Adversarial images can even fool the internal representation of images by neural networks [3]

. That problem has divided the Machine Learning community, with some hailing it as a “deep flaw” of deep neural networks

[4]; and others promoting a more cautious interpretation, and showing, for example, that most classifiers are susceptible to adversarial examples [5, 6].

The distortions were originally obtained via an optimization procedure, [1], but it was subsequently shown that adversarial images could be generated using a simple gradient calculation [5]

or using evolutionary algorithms


Despite the controversy, adversarial images surely suggest a lack of robustness, since they are (for humans) essentially equal to correctly classified images. Immunizing a network against those perturbations increases its ability to generalize, a form of regularization [5]

whose statistical nature deserves further investigation. Even the traditional backpropagation training procedure can be improved with adversarial gradient

[7]. Recent work shows that it is possible for the training procedure to make classifiers robust to adversarial examples by using a strong adversary [8]

or defensive distillation


In this paper, we extend previous works on adversarial images for deep neural networks [1], by exploring the pixel space of such images using random perturbations. That framework (Figure 1) allows us to ask interesting questions about adversarial images. Initial skepticism about the relevance of adversarial images suggested they existed as isolated points in the pixel space, reachable only by a guided procedure with complete access to the model. More recent works [5, 10] claim that they inhabit large and contiguous regions in the space. The correct answer has practical implications: if adversarial images are isolated or inhabit very thin pockets, they deserve much less worry than if they form large, compact regions. In this work we intend to shed light to the issue with an in-depth analysis of adversarial image space.

Fig. 1: Fixed-sized images occupy a high-dimensional space spanned by their pixels (one pixel = one dimension), here depicted as a 2D colormap. Left: classifiers associate points of the input pixel space to output class labels, here ‘banana’ (blue) and ‘mushroom’ (red). From a correctly classified original image (a), an optimization procedure (dashed arrows) can find adversarial examples that are, for humans, essentially equal to the original, but that will fool the classifier. Right: we probe the pixel space by taking a departing image (white diamond), adding random noise to it (black stars), and asking the classifier for the label. In compact, stable regions, the classifier will be consistent (even if wrong). In isolated, unstable regions, as depicted, the classifier will be erratic.

Ii Creating adversarial images

Assume we have a pre-trained classifier that, for each input

, corresponding to the pixels of a fixed-sized image, outputs a vector of probabilities

of the image belonging to the class label . We can assign to the label corresponding to the highest probability . Assume further that , for grayscale images, or for RGB images, where and are the lower and upper limits of the pixel scale.

Assume that is the correct label and that we start with , otherwise there is no point in fooling the classifier. We want to add the smallest distortion to , such that the highest probability will no longer be assigned to . The distortions must keep the input inside its space, i.e., we must ensure that . In other words, the input is box-constrained. Thus, we have the following optimization:

subject to

That formulation is more general than the one presented by [1], for it ignores non-essential details, such as the choice of the adversarial label. It also showcases the non-convexity: since is convex, the inequality is clearly concave [11], making the problem non-trivial even if the model were linear in

. Deep networks, of course, exacerbate the non-convexity due to their highly non-linear model. For example, a simple multi-layer perceptron for binary classification could have

, which is neither convex nor concave due to the hyperbolic tangent.

Ii-a Procedure

Training a classifier usually means minimizing the classification error by changing the model weights. To generate adversarial images, however, we hold the weights fixed, and find the minimal distortion that still fools the network.

We can simplify the optimization problem of eq. 1 by exchanging the

inequality for a term in the loss function that measures how adversarial the probability output is:

subject to

where we introduce the adversarial probability target , which assigns zero probability to all but a chosen adversarial label . That formulation is essentially the same of [1], picking an explicit (but arbitrary) adversary label. We stipulate the loss function: the cross-entropy () between the probability assignments; while [1] keep that choice open.

The constant balances the importance of the two objectives. The lower the constant, the more we will minimize the distortion norm. Values too low, however, may turn the optimization unfeasible. We want the lowest, but still feasible, value for .

We can solve the new formulation with any local search compatible with box-constraints. Since the optimization variables are the pixel distortions, the problem size is exactly the size of the network input, which in our case varies from for MNIST [12] to for OverFeat [13]. In contrast to current neural network training, that reaches hundreds of millions of weights, those sizes are small enough to allow second-order procedures, which converge faster and with better guarantees [14]. We chose L-BFGS-B, a box-constrained version of the popular L-BFGS second-order optimizer [15]. We set the number of corrections in the limited-memory matrix to 15, and the maximum number of iterations to 150. We used Torch7 to model the networks and extract their gradient with respect to the inputs [16]. Finally, we implemented a bisection search to determine the optimal value for [17]. The algorithm is explained in detail in the next section.

Ii-B Algorithm

Algorithm 1 implements the optimization procedure used to find the adversarial images. The algorithm is essentially a bisection search for the constant , where in each step we solve a problem equivalent to Eq. 2. Bisection requires initial lower and upper bounds for , such that the upper bound succeeds in finding an adversarial image, and the lower bound fails. It will then search the transition point from failure to success (the “zero” in a root-finding sense): that will be the best . We can use as lower bound, as it always leads to failure (the distortion will go to zero). To find an upper bound leading to success, we start from a very low value, and exponentially increase it until we succeed. During the search for the optimal we use warm-starting in L-BFGS-B to speed up convergence: the previous optimal value found for is used as initial value for the next attempt.

To achieve the general formalism of eq. 1 we would have to find the adversarial label leading to minimal distortion. However, in datasets like ImageNet [18]

, with hundreds of classes, this search would be too costly. Instead, in our experiments, we opt to consider the adversarial label as one of the sources of random variability. As we will show, this does not upset the analyses.

The source code for adversarial image generation and pixel space analysis can be found in https://github.com/tabacof/adversarial.

0:  A small positive value
0:   solves optimization 2
1:  {Finding initial }
3:  repeat
6:  until  in is
7:  {Bisection search}
8:  ,
9:  repeat
12:     if  in is  then
15:     else
17:     end if
18:  until 
19:  return    
Algorithm 1 Adversarial image generation algorithm

Iii Adversarial space exploration


MNIST with logistic regression. The correct labels are self-evident.

(b) MNIST with convolutional network. The correct labels are self-evident.
(c) OverFeat on ImageNet. From left to right, correct labels: ‘Abaya’, ‘Ambulance’, ‘Banana’, ‘Kit Fox’, ‘Volcano’. Adversarial labels for all: ‘Bolete’ (a type of mushroom).
Fig. 2: Adversarial examples for each network. For all experiments: original images on the top row, adversarial images on the bottom row, distortions (difference between original and adversarial images) on the middle row.

In this section we explore the vector space spanned by the pixels of the images to investigate the “geometry” of adversarial images: are they isolated, or do they exist in dense, compact regions? Most researchers currently believe that images of a certain appearance (and even meaning) are contained into relatively low-dimensional manifolds inside the whole space [19]. However, those manifolds are exceedingly convoluted, discouraging direct geometric approaches to investigate the pixel space.

Thus, our approach is indirect, probing the space around the images with small random perturbations. In regions where the manifold is nice — round, compact, occupying most of the space — the classifier will be consistent (even if wrong). In the regions where the manifold is problematic — sparse, discontinuous, occupying small fluctuating subspaces — the classifier will be erratic.

Iii-a Datasets and models

To allow comparison with the results of [1], we employ the MNIST handwritten digits database (10 classes, 60k training and 10k testing images), and the 2012 ImageNet Large Visual Recognition Challenge Dataset (1000 classes, 1.2M+ training and 150k testing images).

For MNIST, [1]

tested convolutional networks and autoencoders. We employ both convolutional networks and a logistic linear classifier. While logistic classifiers have limited accuracy (

7.5% error), their training procedure is convex [11]. They also allowed us to complement the original results [1] by investigating adversarial images in a shallow classifier.

The convolutional network we employed for MNIST/ConvNet consisted of two convolutional layers with 64 and 128 55 filters, two 2

2 max-pooling layers after the convolutional layers, one fully-connected layer with 256 units, and a softmax layer as output. We used ReLU for nonlinearity, and 0.5 dropout before the two last layers. The network was trained with SGD and momentum. Without data augmentation, this model achieves 0.8% error on the test set.

For ImageNet, we used the pre-trained OverFeat network [13], which achieved 4th place at the ImageNet competition in 2013, with 14.2% top-5 error in the classification task, and won the localization competition the same year. [1] employed AlexNet [20], which achieved 1st place at the ImageNet competition in 2012, with 15.3% top-5 error.

On each dataset, we preprocess the inputs by standardizing each pixel with the global mean and standard deviation of all pixels in the training set images.

Figure 2 illustrates all three cases. Original and adversarial images are virtually indistinguishable. The pixel differences (middle row) do not show any obvious form — although a faint “erasing-and-rewriting” effect can be observed for MNIST. Figures (a)a and (b)b also suggest that the MNIST classifiers are more robust to adversarial images, since the distortions are larger and more visible. We will see, throughout the paper, that classifiers for MNIST and for ImageNet have important differences in how they react to adversarial images.

Iii-B Methods

Each case (MNIST/Logistic, MNIST/ConvNet, ImageNet/OverFeat) was investigated independently, by applying the optimization procedure explained in Section II-A. For ImageNet we sampled 5 classes (Abaya, Ambulance, Banana, Kit Fox, and Volcano), 5 correctly classified examples from each class, and sampled 5 adversarial labels (Schooner, Bolete, Hook, Lemur, Safe), totaling 125 adversarial images. For MNIST, we just sampled 125 correctly classified examples from the 10K examples in the test set, and sampled an adversarial label (from 9 possibilities) for each one. All random sampling was made with uniform probability. To sample only correctly classified examples, we rejected the misclassified ones until we accumulated the needed amount. We call, in the following sections, those correctly classified images originals, since the adversarial images are created from them.

The probing procedure consisted in picking an image pair (an adversarial image and its original), adding varying levels of noise to their pixels, resubmitting both to the classifier, and observing if the newly assigned labels corresponded to the original class, to the adversarial class, or to some other class.

We measured the levels of noise () relative to the difference between each image pair. We initially tested a Gaussian i.i.d. model for the noise. For each image , our procedure creates an image where , and and

are the sample mean and variance of the distortion pixels. In the experiments we ranged

from to . To keep the pixel values of within the original range we employ . In practice, we observed that clamping has little effect on the noise statistics.

An i.i.d. Gaussian model discards two important attributes of the distortions: spatial correlations, and higher-order momenta. We wanted to evaluate the relative importance of those, and thus performed an extra round of experiments that, while still discarding all spatial correlations by keeping the noise i.i.d., adds higher momenta information by modeling non-parametrically the distribution of distortion pixels. Indeed, a study of those higher momenta (Table I) suggests that the adversarial distortions has a much heavier tail than the Gaussians, and we wanted to investigate how that affects the probing. The procedure is exactly the same as before, but with , where is an empirical distribution induced by a non-parametric observation of the distortion pixels. In those experiments we cannot control the level: the variance of the noise is essentially the same as the variance of the distortion pixels.

Our main metric is the fraction of images (in %) that keep or switch labels when noise is added to a departing image, which we use as a measure of the stability of the classifier at the departing image in the pixel space. The fraction is computed over a sample of 100 probes, each probe being a repetition of the experiment with all factors held fixed but the sampling of the random noise.

Mean S.D. Skewness

Ex. Kurtosis

TABLE I: Descriptive statistics of the adversarial distortions for the two datasets averaged over the 125 adversarial examples. Pixels values in . Logistic and ConvNet refer to MNIST dataset, OverFeat refers to ImageNet dataset.

Iii-C Results

Fig. 3: Adding Gaussian noise to the images. We perform the probing procedure explained in Section III-B to measure the stability of the classifier boundaries at different points of the pixel space. To escape the adversarial pockets completely we have to add a noise considerably stronger than the original distortion used to reach them in the first place: adversarial regions are not isolated. That is especially true for ImageNet/OverFeat. Still, the region around the correctly classified original image is much more stable. This graph is heavily averaged: each stacked column along the horizontal axis summarizes 125 experiments 100 random probes.

Figure 3 shows that adversarial images do not appear isolated. On the contrary, to completely escape the adversarial pocket we need to add a noise with much higher variance — notice that the horizontal axis is logarithmic — than the distortion used to reach the adversarial image in the first place.

In both networks, the original images display a remarkable robustness against Gaussian noise (Figures (b)b and (f)f), confirming that robustness to random noise does not imply robustness to adversarial examples [6]. That shows that while the adversarial pockets are not exactly isolated, neither are they as well-behaved as the zones that contain the correctly classified samples.

Fig. 4: Adding Gaussian noise to the images. Another view of the probing procedure explained in Section III-B. Contrarily to the averaged view of Figure 3, here each one of the 125 experiments appears as an independent curve along the Experiments axis (their order is arbitrary, chosen to reduce occlusions). Each point of the curve is the fraction of probes (out of a hundred performed) that keeps their class label.
(a) MNIST / logistic regression
(b) MNIST / convolutional network
(c) ImageNet / OverFeat
Fig. 5: For each of the 125 experiments we measure the fraction of the probe images (i.e., departing image + random noise) that stayed in the same class label. Those fractions are then sorted from biggest to lowest along the Experiments axis. The area under the curves indicates the entire fraction of probes among all experiments that stayed in the same class.

The results in Figure 3 are strongly averaged, each data point summarizing, for a given level of noise, the result of 125 experiments: the fraction of images that fall in each label for all five original class labels, all five original samples from each label, and all five adversarial class labels. In reality there is a lot of variability that can be better appreciated in Figure 4. Here each curve alongside the axis experiments represents a single choice of original class label, original sample, and adversarial class label, thus there are 125 curves. (The order of the curves along this axis is arbitrary and chosen to minimize occlusions and make the visualization easier). The graphs show that depending on a specific configuration, the label may be very stable and hard to switch (curves that fall later or do not fall at all), or very unstable (curves that fall early). Those 3D graphs also reinforce the point about the stability of the correctly classified original images.

The results suggest that the classifiers for MNIST are more resilient against adversarial images than ImageNet/OverFeat. Moreover, the shallow MNIST/logistic behaves differently than the deep MNIST/ConvNet, as shown by the the “falling columns” in Figure 3: initially, a small push in MNIST/logistic throws a larger fraction of the adversarial examples back to the correct space. However, at large noise levels, MNIST/logistc saturates with a larger fraction of images still adversarial than MNIST/ConvNet.

Finally, we wanted to investigate how the nature of the noise added affected the experiments. Recall that our i.i.d. Gaussian noise differs from the original optimized distortion in two important aspects: no spatial correlations, and no important higher-order momenta. To explore the influence of those two aspects, we introduced a noise modeled after the empirical distribution of the distortion pixels. This still ignores spatial correlations, but captures higher-order momenta. The statistics of the distortion pixels are summarized in Table I, and reveal a distribution that is considerably heavier-tailed than the Gaussians we have employed so far.

Figure 5 contrasts the effect of this noise modeled non-parametrically after the distortions with the effect of the comparable Gaussian noise (). Each point in the curves is one of the 125 experiments, and represents the fraction of the 100 probe images that stays in the same class as the departing — adversarial or original — image. The experiments where ordered by this value in each curve (thus the order of the experiments in the curves is not necessarily the same). Here the individual experiments are not important, but the shape of the curves: how early and how quickly they fall.

For ImageNet, the curves for the non-parametric noise (dotted lines) fall before the curves for the Gaussian noise (continuous line), showing that, indeed, the heavier tailed noise affects the images more, even without the spatial correlation. In addition, all curves fall rather sharply. This shows that in almost all experiments, either all probes stay in the same label as the original, or all of them switch. Few experiments present intermediate results. This rather bimodal behavior was already present in the curves of Figure 4.

For MNIST, again, the effect is different: Gaussian and heavy-tail noises behave much more similarly and the curves fall much more smoothly.

Iv Conclusion

Our in-depth analysis reinforces previous claims found in the literature [5, 10]: adversarial images are not necessarily isolated, spurious points: many of them inhabit relatively dense regions of the pixel space. This helps to explain why adversarial images tend to stay adversarial across classifiers of different architectures, or trained on different sets[1]: slightly different classification boundaries stay confounded by the dense adversarial regions.

The nature of the noise affects the resilience of both adversarial and original images. The effect is clear in ImageNet/OverFeat, where a Gaussian noise affects the images less than a heavy-tailed noise modeled after the empirical distribution of the distortions used to reach the adversarial images in the first place. An important next step in the exploration, in our view, is to understand the spatial nature of the adversarial distortions, i.e., the role spatial correlations play.

Recent works have attributed susceptibility to adversarial attacks to the linearity in the network [5], but our experiments suggest the phenomenon may be more complex. A weak, shallow, and relatively more linear classifier (logistic regression), seems no more susceptible to adversarial images than a strong, deep classifier (deep convolutional network), for the same task (MNIST). A strong deep model on a complex task seems to be more susceptible. Are adversarial images an inevitable Achilles’ heel of powerful complex classifiers? Speculative analogies with the illusions of the Human Visual System are tempting, but the most honest answer is that we still know too little. Our hope is that this article will keep the conversation about adversarial images ongoing and help further explore those intriguing properties.


We would like to thank Micael Carvalho for his helpful hints and revision; Jonghoon Jin for open access to his OverFeat wrapper111https://github.com/jhjin/overfeat-torch; and Soumith Chintala, and the rest of the Torch7 team for all of their help on the mailing list. We thank the Brazilian agencies CAPES, CNPq and FAPESP for financial support.