Deep learning systems are broadly vulnerable to adversarial examples, carefully chosen inputs that cause the network to change output without a visible change to a human (Szegedy14, ; goodfellow2014explaining, ). These adversarial examples most commonly modify each pixel by only a small amount and can be found using a number of optimization strategies such as L-BFGS (Szegedy14, ), Fast Gradient Sign Method (FGSM) (goodfellow2014explaining, ), DeepFool (moosavi2016deepfool, ), Projected Gradient Descent (PGD) (madry2017advexamples, )
, as well as the recently proposed Logit-space Projected Gradient Ascent (LS-PGA)anonymous2018thermometer for discretized inputs. Other attack methods seek to modify only a small number of pixels in the image (Jacobian-based saliency map (papernot2016limitations, )), or a small patch at a fixed location of the image (sharif2016accessorize, ).
Adversarial examples have been shown to generalize to the real world. Kurakin et al. kurakin2016adversarial demonstrated that when printed out, an adversarially constructed image will continue to be adversarial to classifiers even under different lighting and orientations. Athalye et al. 2017synthesizing recently demonstrated adversarial objects which can be 3d printed and misclassified by networks at different orientations and scales. Their adversarial objects are designed to be subtle perturbations of a normal object (e.g. a turtle that has been adversarially perturbed to be classified as a rifle). Another work (sharif2016accessorize, )
showed that one can fool facial recognition software by constructing adversarial glasses. These glasses were targeted in that they could be constructed to impersonate any person, but were custom made for the attacker’s face, and were designed with a fixed orientation in mind. Even more recently, Evtimov et al.(evtimov2017robust, ) demonstrated various methods for constructing stop signs that are misclassified by models, either by printing out a large poster that looks like a stop sign, or by placing various stickers on a stop sign. In terms of defenses there has been substantial work on increasing the adversarial robustness of image models to small perturbations of the input madry2017advexamples ; papernot2016distillation ; tramer2017ensemble ; anonymous2018thermometer .
As seen above, a majority of prior work has focused on attacking with and defending against either small or imperceptible changes to the input. In this work we explore what is possible if an attacker no longer restricts themselves to imperceptible changes. We construct an attack that does not attempt to subtly transform an existing item into another. Instead, this attack generates an image-independent patch that is extremely salient to a neural network. This patch can then be placed anywhere within the field of view of the classifier, and causes the classifier to output a targeted class. Because this patch is scene-independent, it allows attackers to create a physical-world attack without prior knowledge of the lighting conditions, camera angle, type of classifier being attacked, or even the other items within the scene.
This attack is significant because the attacker does not need to know what image they are attacking when constructing the attack. After generating an adversarial patch, the patch could be widely distributed across the Internet for other attackers to print out and use. Additionally, because the attack uses a large perturbation, the existing defense techniques which focus on defending against small perturbations may not be robust to larger perturbations such as these. Indeed recent work has demonstrated that state-of-the art adversarially trained models on MNIST are still vulnerable to larger perturbations than those used in training either by searching for a nearby adversarial example using a different metric for distance (chen2017madry, ), or by applying large perturbations in the background (anonymous2018adversarial, ).
The traditional strategy for finding a targeted adversarial example is as follows: given some classifier , some input , some target class and a maximum perturbation , we want to find the input that maximizes , subject to the constraint that . When is parameterized by a neural network, an attacker with access to the model can perform iterated gradient descent on in order to find a suitable input . This strategy can produce a well camouflaged attack, but requires modifying the target image.
Instead, we create our attack by completely replacing a part of the image with our patch. We mask our patch to allow it to take any shape, and then train over a variety of images, applying a random translation, scaling, and rotation on the patch in each image, optimizing using gradient descent. In particular for a given image , patch , patch location , and patch transformations (e.g. rotations or scaling) we define a patch application operator which first applies the transformations to the patch , and then applies the transformed patch to the image at location (see figure 2).
To obtain the trained patch we use a variant of the Expectation over Transformation (EOT) framework of Athalye et al. (2017synthesizing, ). In particular, the patch is trained to optimize the objective function
where is a training set of images, is a distribution over transformations of the patch, and is a distribution over locations in the image. Note that this expectation is over images, which encourages the trained patch to work regardless of what is in the background. This departs from most prior work on adversarial perturbations in the fact that this perturbation is universal in that it works for any background. Universal perturbations were identified in (moosavi2016universal, ), but these required changing every pixel in the image and results were not given in the physical world.
We also consider camouflaged patches which are forced to look like a given starting image. Here we simply add a constraint of the form to the patch objective. This will force the final patch to be within in the norm of some starting patch .
We believe that this attack exploits the way image classification tasks are constructed. While images may contain several items, only one target label is considered true, and thus the network must learn to detect the most "salient" item in the frame. The adversarial patch exploits this feature by producing inputs much more salient than objects in the real world. Thus, when attacking object detection or image segmentation models, we expect a targeted toaster patch to be classified as a toaster, and not to affect other portions of the image.
3 Experimental Results
To test our attack, we compare the efficacy of two whitebox attacks, a blackbox attack, and a control patch. The white box ensemble attack jointly trains a single patch across five ImageNet models: inceptionv3, resnet50, xception, VGG16, and VGG19. We then evaluate the attack by averaging the win rate across all five models. The white box single model attack does the same but only trains and evaluates on a single model. The blackbox attack jointly trains a single patch across four of the ImageNet models, and then evaluates the blackbox attack on a fifth model, which we did not access during training. The control is a picture of a toaster.
During training and evaluation, the patches are rescaled and then digitally inserted on a random location on a random ImageNet image. Figure 2 shows the results.
Note that the patch size required to reliably fool the model in this universal setting (black box, on a targeted class, and over all images, locations and transformations) is significantly larger than those required to perform a non-targeted attack on a single image and a single location in the whitebox setting. For example, Su et al. (one_pixel, ) recently demonstrated that modifying 1 pixel on a 32x32 pixel CIFAR-10 image (0.1% of the pixels in the image) suffices to fool the majority of images with a non-targeted, non-universal whitebox attack. However, our attacks are still far more effective than naively inserting an image with the target class, as is shown in Figure 2 by the relatively poor performance of inserting a real toaster into the scene.
Any of the attacks shown in Figure 2 can be camouflaged in order to reduce their saliency to a human observer. We create a disguised version of the patch by minimizing its L2 distance to a tie-dye pattern and applying a peace sign mask during training. The results from these experiments are found in Figure 3.
In our final experiment, we test the transferability of our attack into the physical world. We print the generated patch with a standard color printer, and put it a variety of real world situations. The results shown in Figure 1 demonstrate that the attack successfully fools the classifier, even when there are other objects within the scene. For a full video demonstration of the attack, see https://youtu.be/i1sp4X57TL4.
We also tested the black box + physical world effectiveness of the patch on the third party Demitasse application111https://itunes.apple.com/us/app/demitasse-image-recognition-cam/id1138211169?mt=8 and found some transferability of the patch but only when the patch takes up a significant fraction of the image. We did not optimize the patch for print-ability as in sharif2016accessorize , which perhaps explains why the patch is not as effective as in Figure 3, which tests black box for different models and not in the physical world. We invite curious readers to try the patch out for themselves by printing out this paper and using the patch in the Appendix.
We show that we can generate a universal, robust, targeted patch that fools classifiers regardless of the scale or location of the patch, and does not require knowledge of the other items in the scene that it is attacking. Our attack works in the real world, and can be disguised as an innocuous sticker. These results demonstrate an attack that could be created offline, and then broadly shared.
There has been substantial work on defending against small perturbations to natural images, at least partially motivated by security concerns papernot2016distillation ; madry2017advexamples ; anonymous2018thermometer . Part of the motivation of this work is that potential malicious attackers may not be concerned with generating small or imperceptible perturbations to a natural image, but may instead opt for larger more effective but noticeable perturbations to the input - especially if a model has been designed to resist small perturbations.
Many ML models operate without human validation of every input and thus malicious attackers may not be concerned with the imperceptibility of their attacks. Even if humans are able to notice these patches, they may not understand the intent of the patch and instead view it as a form of art. This work shows that focusing only on defending against small perturbations is insufficient, as large, local perturbations can also break classifiers.
-  Anonymous. Adversarial spheres. International Conference on Learning Representations, 2018.
-  Anonymous. Thermometer encoding: One hot way to resist adversarial examples. International Conference on Learning Representations, 2018.
-  A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok. Synthesizing robust adversarial examples. arXiv preprint arXiv:1707.07397, 2017.
-  I. Evtimov, K. Eykholt, E. Fernandes, T. Kohno, B. Li, A. Prakash, A. Rahmati, and D. Song. Robust physical-world attacks on deep learning models. arXiv preprint arXiv:1707.08945, 2017.
-  I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
-  S. K. Jiawei Su, Danilo Vasconcellos Vargas. One pixel attack for fooling deep neural networks. arXiv preprint arXiv:1710.08864, 2017.
-  A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533, 2016.
-  A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial examples. arXiv preprint arXiv:1706.06083, 2017.
-  S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard. Universal adversarial perturbations. arXiv preprint arXiv:1610.08401, 2016.
-  S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In , pages 2574–2582, 2016.
-  N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami. The limitations of deep learning in adversarial settings. In Security and Privacy (EuroS&P), 2016 IEEE European Symposium on, pages 372–387. IEEE, 2016.
-  N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In Security and Privacy (SP), 2016 IEEE Symposium on, pages 582–597. IEEE, 2016.
-  M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 1528–1540. ACM, 2016.
-  Y. Sharma and P.-Y. Chen. Breaking the Madry Defense model with L1-based adversarial examples. arXiv preprint arXiv:1710.10733, 2017.
-  C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
-  F. Tramér, A. Kurakin, N. Papernot, D. Boneh, and P. McDaniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint arXiv:1705.07204, 2017.