Sabour et al. (2017)
show that the discriminative performance of a capsule network can be improved by adding another network that reconstructs the input image from the pose parameters and the identity of the correct top-level capsule. Derivatives back-propagated through the reconstruction network force the pose parameters of the top-level capsule to capture a lot of information about the image. A capsule network trained with such a regularizer can output not only a classification, but also a class conditional reconstruction of the input. We show that the reconstruction sub-network can be used as a very effective way to detect adversarial attacks: we reconstruct the input from the identity and pose parameters of the winning top-level capsule to verify that the network is perceiving what we expect it to perceive in a typical example of that class. We propose DARCCC which is an attack independent detection technique relying on the difference between the distribution of class reconstruction distances for genuine images vs adversarial images. We extend DARCCC to more standard image classification networks (convolution neural networks) and we show the effectiveness of our detection method against black box attacks and typical white box attacks on three image data-sets; MNIST, Fashion-MNIST and SVHN.
Our detection method can be defeated by a stronger white-box attack that uses a method (R-BIM) that takes the reconstruction error into account and iteratively perturbs the image so as to allow good reconstruction. However, this stronger attack does not produce typical adversarial images that look like the original image but with a small amount of added noise. Instead, in order to make the model classify the image incorrectly, the perturbation to the original image must be substantial and typically leads to an ”adversarial” image that actually resembles other images of the target class. Moreover, for a capsule network, if enough weight is put on the reconstruction error to avoid detection, it is often impossible to change the image in a way that causes the desired misclassification.
Biggio et al. (2013)
introduced the adversaries for Machine Learning systems. Imperceptible adversarial images for Deep Neural Networks were introduced bySzegedy et al. (2013) where they used a second order optimizer. Fast Gradient Sign method (Goodfellow et al. ) showed that by taking an step in the direction of the gradient, , one can change the label of the input image . Such adversarial attacks which have access to the attacked model are called “white box” attacks. Goodfellow et al. also showed the effectiveness of “black box” attacks where the adversarial images for a model is used to attack another model. Basic Iterative Method (Kurakin et al. (2016)) takes multiple -wide FGSM steps in the ball of the original image.
Kurakin et al. (2018) provides an overview of the oscillating surge of attacks and defenses. Recently several generative approaches are proposed (Samangouei et al. (2018); Ilyas et al. (2017); Meng and Chen (2017)) which assume adversarial images does not exist on the input image manifold. Carlini and Wagner (2017) depicts failure of such adversarial detection techniques. Jetley et al. (2018) and Gilmer et al. (2018) investigate relation of adversarial images to the accuracy of the model and to the input manifold. Since our method conditions on the prediction of the model for generating an image it does not depend on this assumptions. Most recently, Schott et al. (2018) investigated effectiveness of a class conditional generative model as a defense mechanism for MNIST digits. Our method in comparison, does not increase the computational overhead of the classification.
Sabour et al. (2015) shows that adversaries exist for a network with random weights. Therefore, susceptibility to adversarial attacks is not caused by learning and the convolution neural network architectures are inherently fragile. Capsule networks (Sabour et al. (2017); Hinton et al. (2018)
) are a new neural network architecture where neurons activate based on agreement of incoming vectors and defer architecturally from Convolutional neural networks. This new architecture has been proven to be more robust to white box attacks while being as weak as CNNs in defending black box attacks. In this work we address this shortcoming by introducing an adversarial detection mechanism based on reconstruction sub-network of CapsNets. Furthermore, we extend this technique to typical CNNs.
The reconstruction network of the CapsNet proposed in Sabour et al. (2017) takes in the pose parameters of all the class capsules and mask all values to 0 except for the pose parameters of the predicted class. During training they optimize the distance of input image and reconstruction along side the classification loss. We use the same reconstruction network for detecting adversarial attacks by measuring the euclidean distance between the input and a prediction reconstruction. Fig. 2 shows a sample histogram of distances for natural images vs adversarial images. We leverage the difference between the two mentioned distributions and propose DARCCC for detecting adversaries based on the reconstruction from classification. DARCCC distinguishes adversaries by thresholding images based on their reconstruction distance. Fig. 1 shows the reconstructions from real and adversarial data; the deviation of the adversarial reconstructions from input image motivates this approach.
Although the system above is designed for informative pose parameters of the CapsNet, the strategy can be extended beyond CapsNets. We create a similar architecture, “Masked CNN+R”, by using a standard CNN and dividing the penultimate hidden layer into groups corresponding each class. The sum of each neuron group serves as the logit for that particular class and the group itself is passed to the reconstruction sub-network via the same masking operation used bySabour et al. (2017). We also study the role of class conditional reconstruction by omitting the masking and experimenting with a typical “CNN+R” model whose entire penultimate layer is used for reconstruction.
3.1 Detection threshold
We find the threshold for DARCCC based on the expected distance between a validation input image and its reconstruction. If the distance between the input and the reconstruction is above the chosen threshold DARCCC classifies the data as adversarial. Choosing the threshold poses a trade off between false positive and false negative detection rates. Therefore, it should be chosen based on the assumed likelihood of the system being attacked. Such a trade off is discussed by Gilmer et al. (2018). In our experiments we don’t tune this parameter to attacks and set it as the 95th percentile of validation distances. This means our false positive rate on real validation data is 5%.
The three models, Capsule, CNN+R, and Masked CNN+R, are designed to have the same number of parameters. Fig. A.1
shows the architecture we use for each one. For our experiments, all were trained with the same Adam optimizer and for the same number of epochs. We did not do an exhaustive parameters search on these models, instead we chose hyper-parameters that allowed each model to perform roughly equivalently on the test sets. Tab.1 shows the test accuracy of these trained models on the three datasets in our experiments, MNIST (LeCun et al., 1998), FashionMNIST (Xiao et al., 2017), and SVHN (Netzer et al., 2011).
|Dataset||Capsule Model||CNN+R Model||Masked CNN+R Model|
4.1 Black box adversarial attack detection
To test DARCCC on a black box attack, we trained a standard CNN with two layers of convolutions and 2 hidden layers without the aforementioned reconstruction network, and used it to create adversarial attacks using the Fast Gradient Sign Method. Fig. 3 plots the error rate, the attack detection rate, and the successful attack detection rate for each of the 3 models over varying . For all 3 models, DARCCC not only accurately detects the successful attacks (Successful Attack Detection Rate, attacks which changed the networks classification), it detects perturbations regardless of if they changed the networks classification as well (Attack Detection Rate).
4.2 White box adversarial attack detection
We tested DARCCC against white box Basic Iterative Method adversarial attacks targeting each class. We use and . We also clipped the result to be between 0 and 1. The success rate of the attack (flipping the classification to the target class), the attack detection rate (whether the image is tampered with), and the successful attack detection rates (detecting images whose prediction has flipped) are plotted in Fig. 4 for all three models and for the 3 data sets as a function of the number of steps. For all models, DARCCC is able to detect attacks to some degree for Fashion MNIST and MNIST, but on the capsule model it is able to detect adversaries on SVHN as well.
4.2.1 Reconstructive BIM attack
Targeted BIM takes gradient steps to maximize the classification probability of the target class. Since the reconstruction distance is also differentiable we modify BIM into R-BIM which additionally minimizes the reconstruction distance. R-BIM is designed specifically to break DARCCC.
|R-BIM to 0 for Capsule|
|R-BIM to 0 for Masked-CNN+R|
|R-BIM to 0 for CNN+R|
Fig. 5 visualizes the initial input and the result of 100 steps of R-BIM with a target class of ‘0’ for 10 random SVHN images. We see that indeed several of the crafted examples look like ‘0’s. Effectively they are not adversarial images at all since they resemble their predicted class to the human eye. This implies that the gradient is aligned with the true data manifold. Similar visualizations for MNIST and fashion-MNIST can be found in the appendix. For Fashion-MNIST only the capsule model attacks resemble true images from the target class. We still report the same detection rate plots as above in Fig. 6 for R-BIM. Notably R-BIM is significantly less successful than a standard BIM attack in changing the classification. The capsule model in particular exhibits significant resilience to this attack.
We have presented DARCCC, a simple architectural extension that enables adversarial attack detection. DARCCC notably relies on a similarity metric between the reconstruction and the input. This metric is required both during training in order to train the reconstruction network and during test time in order to flag adversarial examples. In the 3 data sets we have evaluated, the distance between examples roughly correlates with semantic similarity. This however is not the case for images in more complex data set such as Cifar10 or ImageNet, in which two images may be similar in terms of content or look, but have significantdistance. This issue will need to be resolved for this method to scale up to more complex problems, and offers a promising avenue for future research.
Notably DARCCC does not rely on a specific predefined adversarial attack. We have shown that by reconstructing the input from the internal class-conditional representation, our system is able to accurately detect black box and white box FGSM and BIM attacks. Of the three models we explored, we showed that the capsule model was the best fitted for this task, and was able to detect adversarial examples with greater accuracy on all the data-sets we explored. We then proposed a new, stronger attack to beat our defense - the Reconstructive BIM attack - in which the adversary optimizes not only the classification loss but also the reconstruction loss. We showed that this attack was less successful than a standard attack, and in particular the capsule model showed great resilience. For more complicated data-sets such as SVHN we showed that the detection method was not able to detect the strong adversarial attacks, but when we visualized the perturbed images they typically appeared to be on the true data manifold and from the target class, so they lacked the paradoxical property of typical adversarial attacks.
ACKNOWLEDGEMENT. We thank Mohammad Norouzi and Nicolas Papernot for their feedback, insight and support.
- Biggio et al. (2013) Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases, pages 387–402. Springer, 2013.
Carlini and Wagner (2017)
Nicholas Carlini and David Wagner.
Adversarial examples are not easily detected: Bypassing ten detection
Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 3–14. ACM, 2017.
- Gilmer et al. (2018) Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S Schoenholz, Maithra Raghu, Martin Wattenberg, and Ian Goodfellow. Adversarial spheres. arXiv preprint arXiv:1801.02774, 2018.
- (4) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples (2014). arXiv preprint arXiv:1412.6572.
- Hinton et al. (2018) Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. 2018.
- Ilyas et al. (2017) Andrew Ilyas, Ajil Jalal, Eirini Asteri, Constantinos Daskalakis, and Alexandros G Dimakis. The robust manifold defense: Adversarial training using generative models. arXiv preprint arXiv:1712.09196, 2017.
- Jetley et al. (2018) Saumya Jetley, Nicholas A Lord, and Philip HS Torr. With friends like these, who needs adversaries? arXiv preprint arXiv:1807.04200, 2018.
- Kurakin et al. (2016) Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533, 2016.
- Kurakin et al. (2018) Alexey Kurakin, Ian Goodfellow, Samy Bengio, Yinpeng Dong, Fangzhou Liao, Ming Liang, Tianyu Pang, Jun Zhu, Xiaolin Hu, Cihang Xie, et al. Adversarial attacks and defences competition. arXiv preprint arXiv:1804.00097, 2018.
- LeCun et al. (1998) Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
- Meng and Chen (2017) Dongyu Meng and Hao Chen. Magnet: a two-pronged defense against adversarial examples. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pages 135–147. ACM, 2017.
Netzer et al. (2011)
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y
Reading digits in natural images with unsupervised feature learning.
NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 5, 2011.
- Sabour et al. (2015) Sara Sabour, Yanshuai Cao, Fartash Faghri, and David J Fleet. Adversarial manipulation of deep representations. arXiv preprint arXiv:1511.05122, 2015.
- Sabour et al. (2017) Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in Neural Information Processing Systems, pages 3856–3866, 2017.
- Samangouei et al. (2018) Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan: Protecting classifiers against adversarial attacks using generative models. arXiv preprint arXiv:1805.06605, 2018.
- Schott et al. (2018) Lukas Schott, Jonas Rauber, Wieland Brendel, and Matthias Bethge. Robust perception through analysis by synthesis. arXiv preprint arXiv:1805.09190, 2018.
- Szegedy et al. (2013) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
- Xiao et al. (2017) Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
Appendix A Architecture
Fig. A.1 shows the architecture of the Capsule network and the CNN+R model used for experiments on MNIST, Fashion-MNIST and SVHN. MNSIT and Fashion-MNIST have exactly same architectures while for SVHN experiments we use larger models. Note that the only difference between the CNN+R and the Masked CNN+R is the masking procedure on the input to the reconstruction network based on the predicted class. All three models have the same number of parameters for each dataset.
Appendix B Histogram of distances
Fig. B.2 and Fig. B.3 visualize the histogram of euclidean distances for real Fashion-MNIST and SVHN validation images (blue) vs the white box FGSM with adversarial images (green) as a proof of concept and motivation. We do not factor the distribution of adversarial distances for picking DARCCC threshold. The threshold is solely based on the validation distances.