Deep neural networks have achieved state-of-the-art performance on a variety of tasks, including image classification, object detection, speech recognition and machine translation. However, they have been shown to be vulnerable to adversarial examples. This incurs a security risk when DNNs are applied to sensitive areas such as finance, medicine, criminal justice and transportation. Adversarial examples are inputs to machine learning models that an attacker constructs intentionally to fool the model. Szegedy et al.  observed that a visually indistinguishable perturbation in pixel space to the original image can alter the prediction of a neural network. Later, a series of papers [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16] designed more sophisticated methods for the worst-case perturbation within a restricted set, often a small ball with .
While a line of work tries to explain why adversarial examples exist [3, 17, 18, 19], a comprehensive analysis of underlying reasons has so far been an open problem, mainly because deep neural networks have complex function forms that a complete mathematical analysis is difficult to achieve. On the other hand, there has been a growing interest in developing tools for tackling the black-box nature of neural networks, among which feature attribution is a widely studied approach [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32]
. Given a predictive model, such a method outputs, for each instance to which the model is applied, a vector of importance scores associated with the underlying features. Feature attribution has been used to improve transparency and fairness of machine learning models[23, 30].
In this paper, we investigate the application of feature attribution to detecting adversarial examples. In particular, we observe that the feature attribution map of an adversarial example near the boundary always differs from that of the corresponding original example. A motivating example is shown in Figure 1, which demonstrates images in CIFAR-10 to be fed into a residual neural network and the corresponding feature attribution from Leave-One-Out (LOO) . The latter interprets decisions from a neural model by observing the effects on the model of erasing each pixel of input before and after the worst-case perturbation by C&W attack. While the perturbation on the original image is visually imperceptible, the feature attribution is altered drastically. We further observe that the difference can be summarized by simple statistics that characterize feature disagreement, which are capable of distinguishing adversarial examples from natural images. We conjecture that this is because adversarial attacks tend to perturb samples into an unstable region on the decision surface.
The above observation led to an effective method for detecting adversarial examples near the decision boundary. On the other hand, there also exists adversarial examples in which the model has high confidence . Previous work has observed several state-of-the-art detection methods are vulnerable to such attacks [33, 34]. However, we observe an interesting phenomenon: middle layers of neural networks still contain information of uncertainty even for high-confidence adversarial examples. Based on this observation, we generalize our method to incorporate multi-layer feature attribution, where attribution scores for intermediate layers are computed without incurring extra model queries.
In numerical experiments, our method achieves superior performance in detecting adversarial examples generated from popular attack methods on MNIST, CIFAR-10 and CIFAR-100 among state-of-the-art detection methods. We also show the proposed method is capable of detecting mixed-confidence adversarial examples, transferring between adversarial examples of different confidence levels, and adversarial examples generated by various types of attacks.
2 Related Work
In this section, we review related work in feature attribution, adversarial attack, adversarial defense and detection.
A variety of methods have been proposed to assign feature attribution scores. For each specific instance where the model is applied, an attribution method assigns an importance score for each feature, by approximating the target model via a linear model locally around the instance. One popular class of methods assumes the differentiability of the model, and propagates the prediction to features through gradients. Examples include direct use of gradient (Saliency Map) , Layer-wise Relevance Propagation (LWRP)  and its improved version DeepLIFT , and Integrated Gradients .
Another class is perturbation-based and thus model-agnostic. Given an instance, multiple perturbed samples are generated by masking different groups of features with a pre-specified reference value. The feature attribution of the instance is computed according to the prediction scores of a model on these samples. Popular perturbation based methods include Leave-One-Out [35, 27], LIME  and KernelSHAP .
It has been observed in Ghorbani et al.  that gradient-based feature attribution maps are sensitive to small perturbations. Adversarial attack to feature attribution is designed to characterize the fragility. On the contrary, robustness of an attribution method has been observed on a robust model. In fact, Yeh et al.  observed that gradient based explanations of an adversarially trained network are less sensitive, and Chalasani et al. 
established theoretical results for the robustness of attribution map on an adversarially trained logistic regression. These observations indicate that the sensitivity of a feature attribution might be rooted in the sensitivity of the model, instead of the attribution method. This motivates the detection of adversarial examples via attribution methods.
Adversarial attacks try to alter, with minimal perturbation, the prediction of an original instance from a given model, which leads to adversarial examples. Adversarial examples can be categorized as targeted or untargeted, depending on whether the goal is to classify the perturbed instance into a given target class or an arbitrary class different from the correct one. Attacks also differ by the type of distance they use to characterize minimal perturbation.and distances are the most commonly used distances. Fast Gradient Sign Method (FGSM) by Goodfellow et al.  is an efficient method to minimize the distance. Kurakin et al.  and Madry et al.  proposed -PGD (BIM), an iterative version of FGSM, which achieves a higher success rate with a smaller size of perturbation. DeepFool presented by Moosavi-Dezfooli et al.  minimizes distance through an iterative linearization procedure. Carlini and Wagner  proposed effective algorithms to generate adversarial examples for each of the three distances. In particular, Carlini and Wagner 
proposed a loss function that is capable of controlling the confidence level of adversarial examples. The Jacobian-based Saliency Map Attack (JSMA) by is a greedy method for perturbation with
metric. Recently, several black-box adversarial attacks that solely depend on probability scores or decisions have been introduced.Chen et al.  and Ilyas et al. [10, 11] introduced score-based methods using zeroth-order gradient estimation to craft adversarial examples. Brendel et al.  introduced Boundary Attack, as a black-box method to minimize the distance, that does not need access to gradient information and relies solely on the model decision. We demonstrate in our experiments that our method is capable of detecting adversarial examples generated by these attacks, regardless of the distance, confidence level, or whether the gradient information is used.
Adversarial defense and detection
To improve the robustness of neural networks, various approaches have been proposed to defend against adversarial attacks, including adversarial training [3, 4, 8, 39, 40], distributional smoothing 42], generative models , feature squeezing , randomized models [45, 46, 47], and verifiable defense [48, 49]. These defenses often involve modifications in the training process of a model, which often require higher computational or sample complexity , and lead to loss of accuracy .
Complimentary to the previous defending techniques, an alternative line of work focuses on screening out adversarial examples in the test stage without touching the training of the original model. Data transformations such as PCA have been used to extract features from the input and layers of neural networks for adversarial detection [52, 53, 54]. Alternative neural networks are used to classify adversarial and original images [55, 56, 57]. Feinman et al. 
proposed to use kernel density estimate (KD) and Bayesian uncertainty (BU) in hidden layers of the neural network for detection.Ma et al.  observed Local Intrinsic Dimension (LID) of hidden-layer outputs differ between the original and adversarial examples. Lee et al. 
obtained the class conditional Gaussian distributions with respect to lower-level and upper-level features of the deep neural network under Gaussian discriminant analysis, which result in a confidence score based on the Mahalanobis distance (MAHA), followed by a logistic regression model on the confidence scores to detect adversarial examples. Through vast experiments, we show that our method achieves comparable or superior performance than these detection methods across various attacks. Furthermore, we show that our method achieves competitive performance for attacks with a varied confidence level, a setting where the other detection methods fail to work[33, 34].
Most related to our work, Tao et al. 
proposed to identify neurons critical for individual attributes to detect adversarial examples, but their method is restricted to models in face recognition. Instead, our method is applicable across different types of image data.Zhang et al.  proposed to identify adversarial perturbations by training a neural network on the saliency map of inputs. However, their method depends on additional neural networks, which are vulnerable to white-box attacks when attackers perturb the image to fool the original model and the new neural network simultaneously.
3 Adversarial detection with feature attribution
3.1 Feature attribution before and after perturbation
Assume that the model is a function which maps an image of dimension to a probability vector of dimension , where is the number of classes. A feature attribution method maps an input image to an attribution vector of the same shape as the image: , such that the th dimension of is the contribution of feature in the prediction of the model on the specific image . We suppress the dependence of on the model for notational convenience. We focus on the leave-One-Out (LOO) method [35, 27] throughout the paper, which assigns to each feature the reduction in the probability of the selected class when the feature in consideration is masked by some reference value, e.g. . Denoting the example with the th feature masked by as , LOO defines as
Adversarial attacks aim to change the prediction of a model with minimal perturbation of a sample, so that human is not able to detect the difference between an original image and its perturbed version . Yet we observed that is sensitive to the small difference between and . Figure 1 shows the attribution maps with the original image and its adversarially perturbed counterpart by C&W attack. Even with human eyes, we can observe an explicit difference in the attribution maps of the original and adversarial images. In particular, adversarial images have a larger dispersion in its importance scores, as demonstrated in Figure 1. We comment here that our proposed framework of adversarial detection via feature attribution is generic to popular feature attribution methods. As an example, we show the performance of Integrated Gradients  for adversarial detection in Appendix 6.1. LOO achieves the best performance among all attribution methods across different data sets.
3.2 Quantify the dispersion in feature attribution maps
Motivated by the apparent differences in the distributions of importance scores between the original and adversarial images, as demonstrated in Figure 1
, we propose to use measures of statistical dispersion in feature attribution to detect adversarial examples. In particular, we tried standard deviation (STD), median absolute deviation (MAD), which is the median of absolute differences between entries and their median, and interquartile range (IQR), which is the difference between the 75th percentile and the 25th percentile among all entries of:
We observe there is a larger dispersion, which we call feature disagreement, between feature contribution to a model for an adversarially perturbed image. The difference is universal across different images. Figure 2 compares the histograms of these three dispersion measures of feature attributions for ResNet on natural test images from CIFAR-10 with those on adversarially perturbed images, where the adversarial perturbation is carried out by C&W Attack. We can see there is a significant difference in the distributions of STD, MAD and IQR between natural and adversarial images. A majority of adversarially perturbed images have a larger dispersion in feature attribution than an arbitrary natural image, besides the corresponding original images. We propose to distinguish adversarial images from natural images by thresholding the IQR of feature attribution maps. In Appendix 6.2, we show the ROC curves of adversarial detection using the three dispersion measures on CIFAR-10 data set with ResNet across three different attacks. All the three measures yield competitive performance. We stick to IQR for the rest of the paper, which is robust and has a slightly superior performance among the three.
3.3 Extension to multi-layer LOO: detection of attacks with mixed confidence levels
Carlini and Wagner  proposed the following objective to generate adversarial images with small perturbation.
maps an image to logits,is the original label, and
is a hyperparameter for tuning confidence. Adversarial images with high confidence can be obtained by assigning a large value to. The loss can be modified to generate constrained perturbation at different confidence levels as well . Recently, Lu et al.  and Athalye et al.  observed that LID has a poor performance when faced with adversarial examples at various confidence scales. In our experiments, a similar phenomenon is observed for several other state-of-the-art detection methods, including KD+BU and MAHA, as is shown in Figure 4. This suggests that characterization of adversarial examples in related work may only hold true for adversarial examples near the decision boundary. IQR of feature attribution map, unfortunately, suffers from the same problem.
To detect adversarial images with mixed confidence levels, we generalize our method to capture dispersion of feature attributions beyond the output layer of the model. For an adversarial example within a small neighborhood of its original example in the pixel space but achieving a high confidence at the output layer in a different class from the original one, the feature representation deviates away from that of its original example gradually along the layers. Thus, we expect neurons of middle layers contain uncertainty that can be captured by a feature attribution map. We denote the map from input to an arbitrary neuron of an intermediate layer of the model by . The feature attribution of neuron is defined as , such that the th entry quantifies the contribution of feature to neuron . For Leave-One-Out (LOO), we have
To coordinate the scale difference between different neurons, we fit a logistic regression for the dispersion of feature attribution from different neurons on a hold-out training set to distinguish adversarial images from original images. The multi-layer extension of our method is called ’ML-LOO’.
We present an experimental evaluation of ML-LOO, and compare our method with several state-of-the-art detection methods. Then we consider the setting where attacks have different confidence levels. We further evaluate the transferability of various detection methods on an unknown attack.
4.1 Known Attacks
We compare our method with state-of-the-art detection algorithms including LID , Mahalanobis (MAHA) , and KD+BU , on three data sets: MNIST, CIFAR-10 and CIFAR-100, with the standard train/test split 
. We used a convolutional network composed of 32-filter convolutional layers followed by a hidden dense layer with 1024 units for MNIST. Each convolutional layer was followed by a max-pooling layer. For both CIFAR-10 and CIFAR-100, we trained a 20-layer ResNet and 121-layer DenseNet  respectively. For each data set, we generated 2,000 adversarial examples from correctly classified test images by each attacking method. Among them, 1,000 adversarial images with the corresponding 1,000 natural images were used for the training process of LID, Mahalanobis and our method. Results are reported for the other 1,000 adversarial images with the corresponding natural images. We consider the following attacking methods, grouped by the norms they are optimized for:
: JSMA .
Let true positive rate (TPR) be the proportion of adversarial images classified as adversarial, and false positive rate (FPR) be the proportion of natural images classified as adversarial. We report area under the curve (AUC) of the ROC curve as the performance evaluation as well as the true positive rates by thresholding FPR at 0.01,0.05 and 0.1, as it is practical to keep misclassified natural images at a low proportion.
The results are reported in Table 1, and the ROC curves on CIFAR-10 with ResNet are shown in Figure 3. The rest of the plots can be found in Appendix 6.3 and Appendix 6.4. ML-LOO shows superior performance over the other three detection methods across different data sets, models for all attacks optimized for and distances. By controlling FPR at 0.1, our method is able to find over 95% adversarial examples generated by most existing attacks.
|C&W MIX||C&W LC||C&W HC|
4.2 Attacks with varied confidence levels
Lu et al.  and Athalye et al.  observed that LID fails when the confidence level of adversarial examples generated from C&W attack varies. We consider adversarial images with varied confidence levels for both and attacks. We use C&W attack for optimizing distance, and adjust the confidence hyperparameter in Equation (3) to achieve mixed confidence levels. To achieve adversarial examples optimized for distance, we use -PGD for optimizing distance, and vary the constraint for different confidence levels.
C&W Attack for optimizing distance
We consider three settings for C&W attack, low-confidence (C&W-LC), mixed-confidence (C&W-MIX) and high-confidence (C&W-HC). We set the confidence parameter for C&W-LC and for C&W-HC. For mixed-confidence C&W attack, we generate adversarial images from C&W attack with the confidence parameter in Equation (3) randomly selected from when generating an adversarial image, so that the distribution of confidence levels for adversarial images is comparable with that of original images. The confidence levels of images under the three settings, along with confidence levels of original images are shown in Figure 4. The confidence level in Figure 4 is defined as , where is the probability score of the predicted class.
We carried out the experiments on ResNet trained on CIFAR-10 using adversarial images generated from the mixed-confidence C&W attack, together with the corresponding original images, as the training data for LID, Mahalanobis, KD+BU, and our method. We test the detection methods on a different set of original and adversarial images generated from three versions: low-confidence C&W attack (), high-confidence C&W attack (), and the mixed-confidence C&W attack. Table 2 (Top) and Figure 4 (Left) show TPRs at different FPR thresholds, AUC, and the ROC curve. Mahalanobis, LID and KD+BU fail to detect adversarial examples of mixed-confidence effectively, while our method performs consistently better for adversarial images across the three settings.
-PGD for optimizing distance
where is the original class, is the cross-entropy loss, and Clip operator clips an image elementwise to an -neighborhood. For mixed-confidence -PGD attack, we generated adversarial images from -PGD with different confidence levels by randomly selecting the constraint in Equation (4) from . The confidence levels of images from mixed-confidence -PGD attack are shown in Figure 4.
We used adversarial images generated from the mixed-confidence -PGD, together with their corresponding original images, as the training data for all detection methods. We report the results on adversarial images generated from three versions: high-confidence -PGD (), low-confidence -PGD (), and the mixed-confidence -PGD that is used to generate the training data. The corresponding original images are different from the training images. Table 2 (Bottom) and Figure 4 (Right) show TPRs at different FPR thresholds, AUC, and the ROC curve. Mahalanobis, LID and KD+BU fail to detect adversarial examples of mixed-confidence effectively, while our method performs significantly better across the three settings.
In this experiment, we evaluate the transferability of different methods by training detection methods on adversarial examples generated from one attacking method and carry out the evaluation on adversarial examples generated from different attacking methods. We trained all methods on adversarial examples generated by C&W attack and carried out the evaluation on adversarial examples generated by the rest of the attacking methods.
Experiments are carried out on MNIST, CIFAR-10, and CIFAR-100 datasets. AUC and TPRs at different FPR thresholds are reported in Table 3. All methods trained on C&W attack are capable of detecting adversarial examples generated from an unknown attack, even when the optimized distance is , or the attack is not gradient-based. The same phenomenon has been observed in Lee et al.  as well. This indicates attacks might share some common features. Our method yields a slightly higher AUC consistently, and has a significantly higher TPR when FPRs are controlled to be small.
In this paper, we have introduced a new framework to detect adversarial examples with multi-layer feature attribution, by capturing the scaling difference of feature attribution scores between original and adversarial examples. We show that our detection method outperforms other state-of-the-art detection methods in detecting various kinds of attacks. In particular, we show our method is able to detect adversarial examples of various confidence levels, and transfers between different attacks.
- Goodfellow et al.  Ian Goodfellow, Nicolas Papernot, Sandy Huang, Yan Duan, and Peter Abbeel. Attacking machine learning with adversarial examples. https://blog.openai.com/adversarial-example-research/, 2017.
- Szegedy et al.  Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
- Goodfellow et al.  Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Representations, 2015.
- Kurakin et al.  Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In International Conference on Learning Representations, 2017.
- Moosavi-Dezfooli et al.  Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In
Papernot et al. [2016a]
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay
Celik, and Ananthram Swami.
The limitations of deep learning in adversarial settings.In 2016 IEEE European Symposium on Security and Privacy (EuroS&P), pages 372–387. IEEE, 2016a.
- Carlini and Wagner  Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pages 39–57. IEEE, 2017.
- Madry et al.  Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Chen et al. 
Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh.
Zoo: Zeroth order optimization based black-box attacks to deep neural
networks without training substitute models.
Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 15–26. ACM, 2017.
- Ilyas et al.  Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In International Conference on Machine Learning, pages 2142–2151, 2018.
- Ilyas et al.  Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Prior convictions: Black-box adversarial attacks with bandits and priors. In International Conference on Learning Representations, 2019.
- Liu et al.  Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
- Papernot et al. [2016b] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07277, 2016b.
- Papernot et al.  Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pages 506–519. ACM, 2017.
- Brendel et al.  Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. In International Conference on Learning Representations, 2018.
- Brunner et al.  Thomas Brunner, Frederik Diehl, Michael Truong Le, and Alois Knoll. Guessing smart: Biased sampling for efficient black-box adversarial attacks. arXiv preprint arXiv:1812.09803, 2018.
- Tanay and Griffin  Thomas Tanay and Lewis Griffin. A boundary tilting persepective on the phenomenon of adversarial examples. arXiv preprint arXiv:1608.07690, 2016.
- Fawzi et al.  Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Analysis of classifiers’ robustness to adversarial perturbations. Machine Learning, 107(3):481–508, 2018.
- Fawzi et al.  Alhussein Fawzi, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Robustness of classifiers: from adversarial to random noise. In Advances in Neural Information Processing Systems, pages 1632–1640, 2016.
- Shrikumar et al.  Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3145–3153. PMLR, 06–11 Aug 2017.
- Bach et al.  Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS One, 10(7):e0130140, 2015.
- Simonyan et al.  Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
- Ribeiro et al.  Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should I trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135–1144. ACM, 2016.
- Li et al. [2016a] Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in nlp. In Proceedings of NAACL-HLT, pages 681–691, 2016a.
- Baehrens et al.  David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How to explain individual classification decisions. Journal of Machine Learning Research, 11:1803–1831, 2010.
- Lipton  Zachary C Lipton. The mythos of model interpretability. arXiv preprint arXiv:1606.03490, 2016.
- Li et al. [2016b] Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220, 2016b.
- Lundberg and Lee  Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pages 4765–4774, 2017.
Štrumbelj and Kononenko 
Erik Štrumbelj and Igor Kononenko.
An efficient explanation of individual classifications using game theory.Journal of Machine Learning Research, 11:1–18, 2010.
- Datta et al.  Anupam Datta, Shayak Sen, and Yair Zick. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. In Security and Privacy (SP), 2016 IEEE Symposium on, pages 598–617. IEEE, 2016.
- Sundararajan et al.  Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3319–3328. JMLR. org, 2017.
- Chen et al.  Jianbo Chen, Le Song, Martin J. Wainwright, and Michael I. Jordan. L-shapley and C-shapley: Efficient model interpretation for structured data. In International Conference on Learning Representations, 2019.
- Lu et al.  Pei-Hsuan Lu, Pin-Yu Chen, and Chia-Mu Yu. On the limitation of local intrinsic dimensionality for characterizing the subspaces of adversarial examples. arXiv preprint arXiv:1803.09638, 2018.
- Athalye et al.  Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International Conference on Machine Learning, pages 274–283, 2018.
- Zeiler and Fergus  Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818–833. Springer, 2014.
- Ghorbani et al.  Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. arXiv preprint arXiv:1710.10547, 2017.
- Yeh et al.  Chih-Kuan Yeh, Cheng-Yu Hsieh, Arun Sai Suggala, David Inouye, and Pradeep Ravikumar. How sensitive are sensitivity-based explanations? arXiv preprint arXiv:1901.09392, 2019.
- Chalasani et al.  Prasad Chalasani, Somesh Jha, Aravind Sadagopan, and Xi Wu. Adversarial learning and explainability in structured datasets. arXiv preprint arXiv:1810.06583, 2018.
- Tramèr et al.  Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. Ensemble adversarial training: Attacks and defenses. In International Conference on Learning Representations, 2018.
- Liu and Hsieh  Xuanqing Liu and Cho-Jui Hsieh. Rob-gan: Generator, discriminator, and adversarial attacker. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2019.
- Miyato et al.  Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. Distributional smoothing with virtual adversarial training. In International Conference on Learning Representations, 2016.
- Papernot et al. [2016c] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (SP), pages 582–597. IEEE, 2016c.
- Song et al.  Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. In International Conference on Learning Representations, 2018.
- Xu et al.  Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. In 25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18-21, 2018, 2018.
- Liu et al.  Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsieh. Towards robust neural networks via random self-ensemble. In Proceedings of the European Conference on Computer Vision (ECCV), pages 369–385, 2018.
- Lecuyer et al.  Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In IEEE Symposium on Security and Privacy, 2019.
- Liu et al.  Xuanqing Liu, Yao Li, Chongruo Wu, and Cho-Jui Hsieh. Adv-bnn: Improved adversarial defense through robust bayesian neural network. In International Conference on Learning Representations, 2019.
- Wong and Kolter  Eric Wong and J Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, 2018.
- Dvijotham et al.  Krishnamurthy Dvijotham, Sven Gowal, Robert Stanforth, Relja Arandjelovic, Brendan O’Donoghue, Jonathan Uesato, and Pushmeet Kohli. Training verified learners with learned verifiers. arXiv preprint arXiv:1805.10265, 2018.
- Schmidt et al.  Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adversarially robust generalization requires more data. In Advances in Neural Information Processing Systems, pages 5019–5031, 2018.
- Tsipras et al.  Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. There is no free lunch in adversarial robustness (but there are unexpected benefits). arXiv preprint arXiv:1805.12152, 2018.
- Li and Li  Xin Li and Fuxin Li. Adversarial examples detection in deep networks with convolutional filter statistics. In Proceedings of the IEEE International Conference on Computer Vision, pages 5764–5772, 2017.
- Bhagoji et al.  Arjun Nitin Bhagoji, Daniel Cullina, Chawin Sitawarin, and Prateek Mittal. Enhancing robustness of machine learning systems via data transformations. In 2018 52nd Annual Conference on Information Sciences and Systems (CISS), pages 1–5. IEEE, 2018.
- Hendrycks and Gimpel  Dan Hendrycks and Kevin Gimpel. Early methods for detecting adversarial images. In International Conference on Learning Representations, 2017.
- Grosse et al.  Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, and Patrick McDaniel. On the (statistical) detection of adversarial examples. arXiv preprint arXiv:1702.06280, 2017.
- Gong et al.  Zhitao Gong, Wenlu Wang, and Wei-Shinn Ku. Adversarial and clean data are not twins. arXiv preprint arXiv:1704.04960, 2017.
- Metzen et al.  Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. In International Conference on Learning Representations, 2017.
- Feinman et al.  Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint arXiv:1703.00410, 2017.
- Ma et al.  Xingjun Ma, Bo Li, Yisen Wang, Sarah M. Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Michael E. Houle, Dawn Song, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In International Conference on Learning Representations, 2018.
- Lee et al.  Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, pages 7167–7177, 2018.
- Tao et al.  Guanhong Tao, Shiqing Ma, Yingqi Liu, and Xiangyu Zhang. Attacks meet interpretability: Attribute-steered detection of adversarial samples. In Advances in Neural Information Processing Systems, pages 7728–7739, 2018.
- Zhang et al.  Chiliang Zhang, Zuochang Ye, Yan Wang, and Zhimou Yang. Detecting adversarial perturbations with saliency. In 2018 IEEE 3rd International Conference on Signal and Image Processing (ICSIP), pages 271–275. IEEE, 2018.
- Chollet et al.  François Chollet et al. Keras. https://github.com/fchollet/keras, 2015.
- He et al.  Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630–645. Springer, 2016.
- Huang et al.  Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
6.1 Performance of Integrated Gradients
In this section, we evaluate the detection of adversarial examples by thresholding the IQR of another popular feature attribution method Integrated Gradients (IG), and compare it with KD+BU, LID, MAHA, and ML-LOO. We consider three attacks FGSM, C&W and JSMA, which are optimized for and distances respectively, on CIFAR-10 dataset with ResNet. We can see that IQR of IG achieves competitive performance in detecting adversarial examples, but not as powerful as the detection methods which incorporated multi-layer information like LID, MAHA and our proposed method ML-LOO. The IG feature is also not as effective as the LOO feature (whose performance is shown in Figure 8).
6.2 Comparison Based on Dispersion Measures
In this section, we compare performance of detection using three different dispersion measures of feature attributions: IQR, STD and MAD.
Figure 7 shows the histograms of these three dispersion measures of feature attributions for ResNet on natural test images from CIFAR-10 with those on adversarially perturbed images, where the adversarial perturbation is carried out by C&W Attack. We can see there is a significant difference in the distributions of the dispersion measures between natural and adversarial images.
Figure 8 shows the ROC curves of the three dispersion statistics on CIFAR-10 with ResNet. We can see that all three dispersion measures achieve competitive performance in detecting adversarial examples generated by three attacks C&W, JSMA and -PGD, but IQR achieves the largest AUC values across all attacking methods.
6.3 ROC curves of detection methods on CIFAR-10, MNIST and CIFAR-100 data sets with FPR from 0.0 to 0.2
Figure 9, Figure 10, Figure 11, Figure 12, and Figure 13 show the ROC curves of four detection method (LID, MAHA, KD+BU, ML-LOO) on three data sets (CIFAR-10, MNIST, CIFAR-100) with three models (CNN, ResNet, DenseNet) under six attacks (FGSM, JSMA, C&W, DeepFool, Boundary, -PGD) where FPR is from 0.0 to 0.2, which is the setting of practical interest. The ROC curves where FPR is from 0.0 to 1.0 are shown in Appendix 6.4.
6.4 ROC curves of detection methods on CIFAR-10, MNIST and CIFAR-100 data sets with FPR from 0.0 to 1.0
In this section, we show the ROC curves of four detection method (LID, MAHA, KD+BU, ML-LOO) on three data sets (CIFAR-10, MNIST, CIFAR-100) with three models (CNN, ResNet, DenseNet) under six attacks (FGSM, JSMA, C&W, DeepFool, Boundary, -PGD) where FPR is from 0.0 to 1.0.