1 Introduction
Those studying deep networks find themselves forced to confront an apparent paradox. On the one hand, there is the demonstrated success of networks in learning class distinctions on training sets that seem to generalise well to unseen test data. On the other, there is the vulnerability of the very same networks to adversarial perturbations that produce dramatic changes in class predictions despite being counterintuitive or even imperceptible to humans. A common understanding of the issue can be stated as follows: “While deep networks have proven their ability to distinguish between their target classes so as to generalise over unseen natural variations, they curiously possess an Achilles heel which must be defended.” In fact, efforts to formulate attacks and counteracting defences of networks have led to a dedicated competition nips2017competition and a body of literature already too vast to summarise in total.
In the current work we attempt to demystify this phenomenon at a fundamental level. We base our work on the geometric decision boundary analysis of moosavidezfooli2018robustness , which we reinterpret and extend into a framework that we believe is simpler and more illuminating with regards to the aforementioned paradoxical behaviour of deep convolutional networks (DCNs) for image classification. Through a fairly straightforward set of experiments and explanations, we clarify what it is that adversarial examples represent, and indeed, what it is that modern DCNs do and do not currently do. In doing so, we tie together work which has focused on adversaries per se with other work which has sought to characterise the feature spaces learned by these networks.
Let represent vectorised input images and
be the average vectorimage over a given dataset. Then, the meannormalised version of the dataset is denoted by
, where the image . We define the perturbation of the image in the direction as: , where is the perturbation scaling factor and is the unitnorm vector in the direction . The image is fed through a network parameterised by and the output score^{2}^{2}2The output of the layer just before the softmax operation, commonly known as the logit layer. for a specific class is given by . This classscore function can be rewritten as , which we equivalently denote by . Our work examines the nature of as a function of movement in specific imagespace directions starting from randomly sampled natural images , for a variety of classification DCNs. With this novel analysis, we uncover three noteworthy observations about these functions that relate directly to the phenomenon of adversarial vulnerability in these nets, all of which are on display in Fig. 1. We now discuss these observations in more detail.Before we begin, we note that these directions are obtained via the method explained in Sec. 3 and by design exhibit either positive or negative association with a specific class. In Fig. 1 we study two such directions for the ‘frog’ class: similar directions exist for all other classes. Firstly, notice that the score of the corresponding class (‘frog’, in this case) as a function of is often approximately symmetrical about some point , i.e. , and monotonic in both halflines. This means that simply increasing the magnitude of correlation between the input image and a single direction causes the net to believe that more (or less) of the class is present. In other words, the imagespace direction sends all images either towards or away from the class . In the former scenario, the direction represents a classspecific universal adversarial perturbation (UAP). Second, let , and let be the projection of onto the space normal to , such that . Then, our results illustrate that there exists a basis of image space containing such that the classscore function is approximately additively separable i.e. for some functions and . This means that the directions under study can be used to alter the nets’ predictions almost independently of each other. However, despite these facts, their 2D visualisation reveals lowlevel structures that are devoid of a clear semantic link to the associated classes, as shown in Fig. 1. Thus, we demonstrate that the learned functions encode a more simplistic notion of class identity than DCNs are commonly assumed to represent, albeit one that generalises to the test distribution to an extent. Unsurprisingly, this does not align with the way in which the human visual system makes use of these data dimensions: ‘adversarial vulnerability’ is simply the name given to this disparity and the phenomena derived from it, with the universal adversarial perturbations of moosavi2017universal being a particularly direct example of this fact.
Finally, we show that nets’ classification performance and adversarial vulnerability are inextricably linked by the way they make use of the above directions, on a variety of architectures. Consequently, efforts to improve robustness by “suppressing” nets’ responses to components in these directions (e.g. gao2017deepcloak ) cannot simultaneously retain full classification accuracy. The features and functions thereof that DCNs currently rely on to solve the classification problem are, in a sense, their own worst adversaries.
2 Related Work
2.1 Fundamental developments in attack methods
Szegedy et al. coined the term ‘adversarial example’ in szegedy2014intriguing , demonstrating the use of boxconstrained LBFGS to estimate a minimal norm additive perturbation to an input image to cause its label to change to a target class while keeping the resulting image within intensity bounds. Strikingly, they locate a smallnorm (imperceptible) perturbation at every point, for every network tested. Further, the adversaries thus generated are able to fool nets trained differently to one another, even when trained with different subsets of the data. Goodfellow et al. goodfellow2015explaining subsequently proposed the ‘fast gradient sign method’ (FGSM) to demonstrate the effectiveness of the local linearity assumption in producing the same result, calculating the gradient of the cost function and perturbing with a fixedsize step in the direction of its sign (optimal under the linearity assumption and an norm constraint). The DeepFool method of MoosaviDezfooli et al. moosavi2016deepfool retains the firstorder framework of FGSM, but tailors itself precisely to the goal of finding the perturbation of minimum norm that changes the class label of a given natural image to any label other than its own. Through iterative attempts to cross the nearest (linear) decision boundary by a tiny margin, this method records successful perturbations with norms that are even smaller than those of goodfellow2015explaining . In moosavi2017universal , MoosaviDezfooli & Fawzi et al. propose an iterative aggregation of DeepFool perturbations that produces “universal” adversarial perturbations: single images which function as adversaries over a large fraction of an entire dataset for a targeted net. While these perturbations are typically much larger than individual DeepFools, they do not correspond to human perception, and indicate that there are fixed imagespace directions along which nets are vulnerable to deception independently of the imagespace locations at which they are applied. They also demonstrate some generalisation over network architectures.
Sabour & Cao et al. sabour2016manipulation pose an interesting variant of the problem: instead of “label adversaries”, they target “feature adversaries” which minimise the distance from a particular guide image in a selected network feature space, subject to a constraint on the norm of imagespace distance from a source image. Despite this constraint, the adversarial image mimics the guide very closely: not only is it nearly always assigned to the guide’s class, but it appears to be an inlier with respect to the guideclass distribution in the chosen feature space. Finally, while “adversaries” are conceived of as small perturbations applied to natural images such that the resulting images are still recognisable to humans, the “fooling images” of Nguyen et al. nguyen2015deep
are completely unrecognisable to humans and yet confidently predicted by deep networks to be of particular classes. Such images are easily obtained by both evolutionary algorithms and gradient ascent, under direct encoding of pixel intensities (appearing to consist mostly of noise) and under CPPN
stanley2007compositional regularised encoding (appearing as abstract midlevel patterns).2.2 Analysis of adversarial vulnerability and proposed defences
In wang2016theoretical , Wang et al. propose a nomenclature and theoretical framework with which to discuss the problem of adversarial vulnerability in the abstract, agnostic of any actual net or attack thereof. They denote an oracle relative to whose judgement robustness and accuracy must be assessed, and illustrate that a classifier can only be both accurate and robust (invulnerable to attack) relative to its oracle if it learns to use exactly the same feature space that the oracle does. Otherwise, a network is vulnerable to adversarial attack in precisely the directions in which its feature space departs from that of the oracle. Under the assumption that a net’s feature space contains some spurious directions, Gao et al. gao2017deepcloak
propose a subtractive scheme of suppressing the neuronal activations (
i.e. feature responses) which change significantly between the natural and adversarial inputs. Notably, the increase in robustness is accompanied by a loss of performance accuracy. An alternative to network feature suppression is the compression of input image data explored in e.g. maharaj2015improving ; das_jpegcompression ; xie_randomisation .Goodfellow et al. goodfellow2015explaining hypothesise that the high dimensionality and excessive linearity of deep networks explain their vulnerability. Tanay and Griffin tanay2016boundary begin by taking issue with the above via illustrative toy problems. They then advance an explanation based on the angle of intersection of the separating boundary with the data manifold which rests on overfitting and calls for effective regularisation  which they note is neither solved nor known to be solvable for deep nets. A variety of trainingbased goodfellow2015explaining ; madry2017towards ; lu2017safetynet ; metzen2017detecting methods are proposed to address the premise of the preceding analyses. Hardening methods goodfellow2015explaining ; madry2017towards investigate the use of adversarial examples to train more robust deep networks. Detectionbased methods lu2017safetynet ; metzen2017detecting
view adversaries as outliers to the training data distribution and train detectors to identify them as such in the intermediate feature spaces of nets. Notably, these methods
lu2017safetynet ; metzen2017detecting have not been evaluated on the feature adversaries of Sabour & Cao et al. sabour2016manipulation . Further, data augmentation schemes such as that of Zhang et al. zhang2018mixup , wherein convex combinations of input images are mapped to convex combinations of their labels, attempt to enable the nets to learn smoother decision boundaries. While their approach zhang2018mixup offers improved resistance to singlestep gradient sign attacks, it is no more robust to iterative attacks of the same type.Over the course of the line of work in moosavidezfooli2018robustness , fawzi2016semirandom , fawzi2017classification , and fawzi2017robustness , the authors build up an imagespace analysis of the geometry of deep networks’ decision boundaries, and its connection with adversarial vulnerability. In fawzi2017classification , they note that the DeepFool perturbations of moosavi2016deepfool tend to evince relatively high components in the subspace spanned by the directions in which the decision boundary has a high curvature. Also, the sign of the mean curvature of the decision boundary in the vicinity of a DeepFooled image is typically reversed with respect to that of the corresponding natural image, which provides a simple scheme to identify and undo the attack. They conclude that a majority of imagespace directions correspond to nearflatness of the decision boundary and are insensitive to attack, but along the remaining directions, those of significant curvature, the network is indeed vulnerable. Further, the directions in question are observed to be shared over sample images. They illustrate in moosavidezfooli2018robustness why a hypothetical network which possessed this property would theoretically be predicted to be vulnerable to universal adversaries, and note that the analysis suggests a direct construction method for such adversaries as an alternative to the original randomised iterative approach of moosavi2017universal : they can be constructed as random vectors in the subspace of shared highcurvature dimensions.
3 Method
The analysis begins as in moosavidezfooli2018robustness , with the extraction of the principal directions and principal curvatures of the classifier’s imagespace class decision boundaries. Put simply, a principal direction vector and its associated principal curvature tell you how much a surface curves as you move along it in a particular direction, from a particular point. Now, it takes many decision boundaries to characterise the classification behaviour of a multiclass net: for a class classifier. However, in order to understand the boundary properties that are useful for discriminating a given class from all others, it should suffice to analyse only the 1vs.all decision boundaries. Thus, for each class , the method proceeds by locating samples very near to the decision boundary between and the union of remaining classes . In practice, for each sample, this corresponds to the decision boundary between and the closest neighbouring class , which is arrived at by perturbing the sample from the latter (“source”) to the former (“target”). Then, the geometry of the decision boundary is estimated as outlined in Alg. 1 below^{3}^{3}3For more discussion about the implementation and associated concepts, refer to Appendix Sec. A.2., closely following the approach of moosavidezfooli2018robustness :
compute eigenvectors and eigenvalues of mean Hessian
returnThe authors of moosavidezfooli2018robustness advance a hypothesis connecting positively curved directions with the universal adversarial perturbations of moosavi2017universal . Essentially, they demonstrate that if the normal section of a net’s decision surface along a given direction can be locally bounded on the outside by a circular arc of a particular positive curvature in the vicinity of a sample image point, then geometry accordingly dictates an upper bound on the distance between that point and the boundary in that direction. If such directions and bounds turn out to be largely common across sample image points (which they do), then the existence of universal adversaries follows directly, with higher curvature implying lowernorm adversaries. This argument is depicted visually in Appendix Sec. A.1. It is from this point that we move beyond the prior art and begin an iterative loop of further analysis, experimentation, and demonstration, as follows.
4 Experiments and Analysis
Provided only that the secondorder boundary approximation holds up well over a sufficiently wide perturbation range and variety of images, the model implies that the distance of such adversaries from the decision boundary should increase as a function of their norm. Also, the attack along any positively curved direction should in that case be associated with the corresponding target class: the class in the call to Alg. 1. And while positively curved directions may be of primary interest in moosavidezfooli2018robustness , the extension of the above geometric argument to the negativecurvature case points to an important corollary: as sufficient steps along positivecurvature directions should perturb increasingly into class , so should steps along negativecurvature directions perturb increasingly away from class . Finally, the plethora of approximately zerocurvature (flat) directions identified in fawzi2017classification ; moosavidezfooli2018robustness should have negligible effect on class identity.
4.1 Class identity as a function of the component in specific imagespace directions
To test how well the above conjectures hold in practice, we graph statistics of the target and nontarget class scores over the dataset as a function of the magnitude of the perturbation applied in directions identified as above. The results are depicted in Fig. 2, in which the predicted phenomena are readily evident. Along the selected positivecurvature directions, as the perturbation magnitude increases (with either sign), the population’s target class score approaches and then surpasses the highest nontarget class score. The monotonicity of this effect is laid bare by graphing the fraction of nontarget samples perturbed into the target class, alongside the median target class softmax score. Note, again, that the link between the directions in question and the target class identity is established a priori by Alg. 1. We continue in the Appendix and show that, as predicted, the same phenomenon is evident in reverse when using negativecurvature directions instead: see Fig. 7. All that changes is that it is the population’s nontarget class scores that overtake its target class score with increasing perturbation norm, with natural samples of the target class accordingly being perturbed out of it. We also illustrate the point that flatness of the decision boundary manifests as flatness of both target and nontarget class scores: over a wide range of magnitudes, these directions do not influence the network in any way. While Fig. 2 illustrates these effects at the level of the population, Fig. 1 shows a disaggregation into individual sample images, with one response curve per sample from a large set. The populationlevel trends remain evident, but another fact becomes apparent: empirically, the shapes of the curves change very little between most samples. They shift vertically to reflect the class score contribution of the orthonormal components, but they themselves do not otherwise much depend on those components. That is to say that at least some key components are approximately additively separable from one another. This fact connects directly to the fact that such directions are “shared” across samples in the first place, and thus identifiable by Alg. 1.
A more intuitive picture of what the networks are actually doing begins to emerge: they are identifying the highcurvature imagespace directions as features associated with respective class identities, with the curvature magnitude representing the sensitivity of class identity to the presence of that feature. But if this is true, it suggests that what we have thus identified are actually the directions which the net relies on generally in predicting the classes of natural images, with the curvaturescumsensitivities representing their relative weightings. Accordingly, it should be possible to disregard the “flat” directions of nearzero curvature without any noticeable change in the network’s class predictions.
4.2 Network classification performance versus effective data dimensionality
To confirm the above hypothesis regarding the relative importance of different imagespace directions for classification, we plot the training and test accuracies of a sample of nets as a function of the subspace onto which their input images are projected. The input subspace is parametrised by a dimensionality parameter , which controls the number of basis vectors selected per class. We use four variants of selection: the most positively curved directions per class (yielding the subspace ); the most negatively curved directions per class (yielding the subspace ; the union of the previous two (subspace ); and the least curved (flattest) directions per class (subspace ). The subspace so obtained is represented by the orthonormalised basis matrix
(obtained by QR decomposition of the aggregated directions), and each input image
is then projected^{4}^{4}4The mean trainingset orthogonal component can be added, but is approximately in practice for data normalised by mean subtraction, as is the case here. onto as . Accuracies on as a function of are shown in the top row of Fig. 3.The outcome is striking: it is evident that in many cases, classification decisions have effectively already been made based on a relatively small number of features, corresponding to the most curved directions. The sensitivity of the nets along these directions, then, is clearly learned purposefully from the training data, and does largely generalise in testing, as seen. Note also that at this level of analysis, it essentially does not matter whether positively or negatively curved directions are chosen. Another important point emerges here. Since it is the highcurvature directions that are largely responsible for determining the nets’ classification decisions, the nets should be vulnerable to adversarial attack along precisely these directions.
4.3 Link between classification and adversarial directions
It has already been noted in fawzi2017classification that adversarial attack vectors evince high components in subspaces spanned by highcurvature directions. We expand the analysis by repeating the procedure of Sec. 4.2 for various attack methods, to determine whether existing attacks are indeed exploiting the directions in accordance with the classifier’s reliance on them. Results are displayed in the bottom row of Fig. 3, and should be compared against the row above. The graphs in these figures illustrate the direct relationship between the fraction of adversarial norm in given subspaces and the corresponding usefulness of those subspaces for classification. The inclusion of the saliency images of simonyan2013visualising alongside the attack methods makes explicit the fact that adversaries are themselves an exposure of the net’s notion of saliency.
By now, two results hint at a simpler and more direct way of identifying bases of classification/adversarial directions. First, a close inspection of the classscore curves sampled and displayed in Fig. 1 reveals a direct connection between the curvature of a direction near the origin and its derivative magnitude over a fairly large interval around it. Second, this observation is made more clear in Fig. 3 where it can be seen that the directions obtained by boundary curvature analysis in Alg. 1 correspond to the directions exploited by various firstorder methods. Thus, we hypothesise that to identify such a basis, one need actually only perform SVD on a matrix of stacked classscore gradients^{5}^{5}5In fact, this analysis is begun in moosavi2017universal , but only the singular values are examined.. Here, we implement this using a collection of DeepFool perturbations to provide the required gradient information, and repeat the analysis of Sec. 4.2, using singular values to order the vectors. The results, in Fig. 4, neatly replicate the previously seen classification accuracy trends for hightolow and lowtohigh curvature traversal of imagespace directions. Henceforth, we use these directions directly, simplifying analysis and allowing us to analyse ImageNet networks.
While Fig. 3 displays the magnitudes of components of precomputed adversarial perturbations in different subspaces, we also design a variation on the analysis to illustrate how effective an efficient attack method (DeepFool) is when confined to the respective subspaces. This is implemented by simply projecting the gradient vectors used in solving DeepFool’s linearised problem onto each subspace before otherwise solving the problem as usual. The results, displayed in Fig. 5, thus represent DeepFool’s “earnest” attempts to attack the network as efficiently as possible within each given subspace. It is evident that the attack must exploit genuine classification directions in order to achieve low norm.
Accuracy ()  Fooling rate (%)  

26798.72  63.96  57.75  100.00  100.00  100.00  100.00  100.00  100.00  
26515.20  53.19  55.80  32.75  77.25  88.95  92.20  94.35  97.65  
26327.03  46.86  53.50  35.55  58.35  77.90  85.95  89.25  95.65  
26159.98  41.92  51.75  36.15  49.80  66.20  76.90  82.95  92.90  
26008.02  37.98  48.10  41.65  49.25  59.95  68.05  74.80  88.30 
4.4 On image compression and robustness to adversarial attack
The above observations have made it clear that the most effective directions of adversarial attack are also the directions that contribute the most to the DCNs’ classification performance. Hence, any attempt to mitigate adversarial vulnerability by discarding these directions, either by compressing the input data maharaj2015improving ; das_jpegcompression ; xie_randomisation or by suppressing specific components of image representations at intermediate network layers gao2017deepcloak , must effect a loss in the classification accuracy. Further, our framework anticipates the fact that the nets must remain just as vulnerable to attack along the remaining directions that continue to determine classification decisions, given that the corresponding classscore functions, which possess the properties discussed earlier, remain unchanged. We use image downsampling as an example data compression technique to illustrate this effect on ImageNet.
We proceed by inserting a preprocessing unit between the DCN and its input at test time. This unit downsamples the input image to a lower size before upsampling it back to the original input size
. The resizing (by bicubic interpolation) serves to reduce the effective dimensionality of the input data. For a randomly selected set of
ImageNet ILSVRC15 test images, we observe the change in classification accuracy over different values of , shown in column 4 of Table 1. The fooling rates^{6}^{6}6Measured as a percentage of samples from the dataset that undergo a change in their predicted label. for the downsampled versions of these natural images’ adversarial counterparts, produced by applying DeepFool to the original network (without the resampling unit), follow in column 5 of the table. At first glance, it appears that the downsamplingbased preprocessing unit has afforded an increase in the network robustness at a moderate cost in accuracy. Results pertaining to this tradeoff have been widely reported maharaj2015improving ; das_jpegcompression ; gao2017deepcloak . Here, we take the analysis a step further.To start, we note the fact that the methodology just described represents a transfer attack from the original net to the net as modified by the inclusion of the resampling unit. As DeepFool perturbations are not designed to transfer in this manner, we first augment them by simply increasing their norm by a scalar factor . We adjust from unity up to a point at which the mean DeepFool perturbation norm is still a couple of orders of magnitude smaller than the mean image norm, such that the perturbations remain largely imperceptible. The corresponding fooling rates grow steadily with respect to , as is observable in Table 1. Hence, although the original fullresolution perturbations may be suboptimal attacks on the resampling variants of the network (as some components are effectively lost to projection onto the compressed space), sufficient rescaling restores their effectiveness. On the other hand, the modified net continues to be equally vulnerable along the remaining effective classification directions, and can easily be attacked directly. To go about this, we simply take the SVD of the stack of downsampled DeepFool perturbations, for values of and (owing to memory constraints). The resulting singular vectors span the entire space of classification/adversarial directions of the corresponding resampling network, as can be seen from the accuracy values in the rightmost subplot of Fig. 4. More crucially, lowernorm DeepFools can be obtained by restricting the attack’s iterative linear optimisation procedure to the space spanned by these compressed perturbations, exactly as described in Sec. 4.3 and displayed in Fig. 5. This subspaceconfined optimisation is analogous to designing a whitebox DeepFool attack for the new network architecture inclusive of the resampling unit, instead of the original network, and is as effective as before. Note that this observation is consistent with the results reported in xie_randomisation , where the strength of the examined gradientbased attack methods increases progressively as the targeted model better approximates the defending model.
5 Conclusion
In this work, we expose a collection of directions along which a given net’s classscore output functions exhibit striking similarity across sample images. These functions are nonlinear, but are de facto of a relatively constrained form: roughly axissymmetric^{7}^{7}7Though not necessarily so for MNIST, because of its constraints: see Appendix Sec. A.4. and typically monotonic over large ranges. We illustrate a close relationship between these directions and class identity: many such directions effectively encode the extent to which the net believes that a particular target class is or is not present. Thus, as it stands, the predictive power and adversarial vulnerability of the studied nets are intertwined owing to the fact that they base their classification decisions on rather simplistic responses to components of the input images in specific directions, irrespective of whether the source of those components is natural or adversarial. Clearly, any gain in robustness obtained by suppressing the net’s response to these components must come at the cost of a corresponding loss of accuracy. We demonstrate this experimentally. We also note that these robustness gains may be lower than they appear, as the network actually remains vulnerable to a properly designed attack along the remaining directions it continues to use. A discussion including some nuanced observations and connections to existing work that follow from our study can be found in Appendix Sec. A.4. To conclude, we believe that for any scheme to be truly effective against the problem of adversarial vulnerability, it must lead to a fundamentally more insightful (and likely complicated) use of features than presently occurs. Until then, those features will continue to be the nets’ own worst adversaries.
Acknowledgements.
This work was supported by the ERC grant ERC2012AdG 321162HELIOS, EPSRC grant Seebibyte EP/M013774/1 and EPSRC/MURI grant EP/N019474/1. We would also like to acknowledge the Royal Academy of Engineering, FiveAI, and extend our thanks to SeyedMohsen MoosaviDezfooli for providing his research code for curvature analysis of decision boundaries of DCNs.
References
 (1) NIPS: 2017 competition on adversarial attacks and defenses. https://www.kaggle.com/nips2017adversariallearningcompetition (2017) accessed: 20180312.
 (2) MoosaviDezfooli*, S.M., Fawzi*, A., Fawzi, O., Frossard, P., Soatto, S.: Robustness of classifiers to universal perturbations: A geometric perspective. In: International Conference on Learning Representations. (2018)
 (3) Lin, M., Chen, Q., Yan, S.: Network in network. International Conference on Learning Representations (2013)

(4)
MoosaviDezfooli*, S.M., Fawzi*, A., Fawzi, O., Frossard, P.:
Universal adversarial perturbations.
In: Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, IEEE (2017) 86–94

(5)
Gao, J., Wang, B., Lin, Z., Xu, W., Qi, Y.:
Deepcloak: Masking deep neural network models for robustness against adversarial samples.
In: International Conference on Learning Representations. (2017)  (6) Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks. In: International Conference on Learning Representations. (2014)
 (7) Goodfellow, I., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: International Conference on Learning Representations. (2015)
 (8) MoosaviDezfooli, S.M., Fawzi, A., Frossard, P.: Deepfool: a simple and accurate method to fool deep neural networks. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Number EPFLCONF218057 (2016)
 (9) Sabour*, S., Cao*, Y., Faghri, F., Fleet, D.J.: Adversarial manipulation of deep representations. In: International Conference on Learning Representations. (2016)
 (10) Nguyen, A., Yosinski, J., Clune, J.: Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2015) 427–436
 (11) Stanley, K.O.: Compositional pattern producing networks: A novel abstraction of development. Genetic programming and evolvable machines 8(2) (2007) 131–162
 (12) Wang, B., Gao, J., Qi, Y.: A theoretical framework for robustness of (deep) classifiers under adversarial noise. arXiv preprint arXiv:1612.00334 (2016)
 (13) Maharaj, A.V.: Improving the adversarial robustness of convnets by reduction of input dimensionality (2015)
 (14) Das, N., Shanbhogue, M., Chen, S., Hohman, F., Chen, L., Kounavis, M.E., Chau, D.H.: Keeping the bad guys out: Protecting and vaccinating deep learning with JPEG compression. CoRR abs/1705.02900 (2017)
 (15) Xie, C., Wang, J., Zhang, Z., Ren, Z., Yuille, A.L.: Mitigating adversarial effects through randomization. CoRR abs/1711.01991 (2017)
 (16) Tanay, T., Griffin, L.: A boundary tilting persepective on the phenomenon of adversarial examples. arXiv preprint arXiv:1608.07690 (2016)

(17)
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.:
Towards deep learning models resistant to adversarial attacks.
International Conference on Learning Representations (2018)  (18) Lu, J., Issaranon, T., Forsyth, D.: Safetynet: Detecting and rejecting adversarial examples robustly. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2017) 446–454
 (19) Metzen, J.H., Genewein, T., Fischer, V., Bischoff, B.: On detecting adversarial perturbations. arXiv preprint arXiv:1702.04267 (2017)
 (20) Zhang, H., Cisse, M., Dauphin, Y., LopezPaz, D.: mixup: Beyond empirical risk minimization. In: International Conference on Learning Representations. (2018)
 (21) Fawzi*, A., MoosaviDezfooli*, S.M., Frossard, P.: Robustness of classifiers: from adversarial to random noise. In: Advances in Neural Information Processing Systems. (2016) 1632–1640
 (22) Fawzi*, A., MoosaviDezfooli*, S.M., Frossard, P., Soatto, S.: Classification regions of deep neural networks. arXiv preprint arXiv:1705.09552 (2017)
 (23) Fawzi, A., MoosaviDezfooli, S.M., Frossard, P.: The robustness of deep networks: A geometrical perspective. IEEE Signal Processing Magazine 34(6) (2017) 50–62
 (24) Simonyan, K., Vedald, A., Zisserman, A.: Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 (2013)
 (25) Russakovsky*, O., Deng*, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., FeiFei, L.: ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 115(3) (2015) 211–252
 (26) do Carmo, M.: Differential Geometry of Curves and Surfaces. PrenticeHall (1976)
 (27) Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on. Volume 1., IEEE (2005) 886–893
Appendix A Appendix
a.1 Illustration of fundamental geometric argument
a.2 Details and clarifications of use of differential geometric concepts
For a more involved treatment of the basic design of the decision boundary curvature analysis algorithm, consult the references moosavidezfooli2018robustness and fawzi2017classification from which it is derived. If still in need of further clarification of fundamental concepts of differential geometry, consult a book such as docarmo1976differential . We now address some specifics not discussed elsewhere, including in our own main text.
For one, it may seem strange to speak of the “curvature” of a decision boundary of a ReLU network, as analytically, this is everywhere either zero or undefined. “Curvature” is here (and in the relevant citations) computed via finitedifference numerical methods, implicitly corresponding to a smooth approximation of the actual piecewise linear boundary.
The astute reader may likewise notice that the use of the terms ‘principal direction’ and ‘principal curvature’ here (as originally in moosavidezfooli2018robustness ) is somewhat relaxed. Let us clarify this point. Strictly speaking, the principal directions at a point on a manifold in an embedding space (such as an dimensional class decision boundary embedded in dimensional image space) are a local concept, forming an orthonormal basis spanning the space tangent to the manifold at that point. The principal curvature associated with each principal direction is the curvature, at the point of tangency, of the normal section of the manifold in the principal direction.
Generally, there is no reason to think that tangent spaces at different points on the boundary surface should coincide, and so a priori, it may not make any sense to speak of “principal directions" in the embedding space. However, the authors of moosavidezfooli2018robustness are fully aware of this, and base their analysis on the hypothesis that there exist imagespace directions which, when projected onto the respective tangent spaces of different points sampled from the boundary, correspond to similar curvature patterns. In other words, they assume that these directions are largely shared across sample images. A curvature can then be associated with each such direction by, for instance, taking the mean of the curvatures measured at those sample points. This is the relaxation of the terminology referred to above: the “principal directions” here represent a rotation of the canonical (dimensional) imagespace axes, and the “principal curvature” associated with each direction represents the sample mean curvature measured along the tangent component of that vector at each sample point in a set.
In practice, the above can in fact be simplified, as is done explicitly in the version of the algorithm given in the main text as Alg. 1. Rather than managing the expense and complexity of the tangentspace projections, it suits our purposes here to work directly with the Hessian of the embedding function, effectively omitting the projection step.
Finally, we note that in contrast to the simplification given in Alg. 1, the (very large) sample mean Hessians are never actually computed and stored explicitly. Instead, for each , a function is constructed that approximates
via finite differences of backpropagated gradients. The MATLAB function
eigs uses to compute the eigendecomposition of the approximated .a.3 Class identity as a function of perturbation scale: negativecurvature and flat directions
a.4 Additional discussion, observations, and relationships with existing work
First, regarding the main result(s) of the paper as summarised in the conclusion (Sec. 5), we would like to point out that the discovery of the universal adversarial perturbations of moosavi2017universal strongly hinted at this outcome in advance. Those attacks are “universal” in precisely the sense that certain fixed directions perturb different images across class boundaries irrespective of the diversity in their individual appearances, i.e., irrespective of the input components in all directions orthogonal to the perturbation. In fact, it is precisely because the functions are as separable as they are in this sense that the method used is able to identify them as well as it does. Note also that our results explain the “dominant label” phenomenon of moosavi2017universal , noted therein as curious but otherwise left unaddressed, in which a given universal perturbation overwhelmingly moves examples into a small number of target classes regardless as to original class identities. This is a manifestation of the targeting of particular classes by particular directions, a phenomenon so strong that it manifests in moosavi2017universal despite the fact that their algorithm never explicitly optimises for this property.
In the context of discussing the adversarial vulnerability of DCNs, we would advise caution in using terms ‘overfitting’ and ‘generalisation’, particularly if done speculatively. Inspection of Figs. 3 (top row) and 4 will convince the reader that the directions of vulnerability produce fits that generalise very well to unseen data generated by the same distribution, observable as the near identity between training and test accuracy curves over the directions of highest curvature (or derivative) magnitude. This does not
correspond to the classical notion of overfitting in machine learning. In fact, overfitting, observable as the divergence between the train and test error curves, happens over less salient (and thus, less vulnerable) directions. This can again be confirmed by inspection. The fact that the nets extract characterisations of their targets that do not correspond to that of humans is a separate issue.
Finally, the approximate symmetry of the feature response functions may appear counterintuitive: why should the additive inverse of a given perturbation pattern produce such a similar result to the original pattern? We suggest that at least part of the explanation may rest in a fact long known in the handengineering of features for computer vision: natural image descriptors must typically neglect sign, because contrast inversion is a fact of the world. For instance, consider how a black bird and a white bird, which share the label bird, differ from one another when both are set against the background of a blue sky. One can revisit, for instance, dalal2005histograms for a reminder. Empirically, this nonlinearity appears to be particularly important. Note that DCNs trained on MNIST are exceptional in that they may not adhere to this, as MNIST is unnatural in its fixing of contrast sign: e.g. the net need never learn that a black vertical stroke against a white background also represents the digit ‘1’.
Comments
There are no comments yet.