. It has been shown that adversarial examples can be created by minimally modifying the original input samples such that a DNN mis-classifies them with high confidence. DNNs are often criticized as black-box models; adversarial examples raise further concerns by highlighting blind spots of DNNs. At the same time, adversarial phenomena provide an opportunity to understand DNN’s behavior to minor perturbations in visual inputs.
Methods that generate adversarial examples either modify each image pixel by a small amount [24, 8, 14, 13] often imperceptible to human vision or few image pixels by a large visible amounts [20, 22, 4, 12, 7]. Pixel attack  changes few image pixels, but it requires small images (e.g., 3232) and does not provide control over noise location. Small noise patches were introduced by  in the form of glasses to cover human face to deceive face recognition systems. Similarly, Evtimov  added noise patches as rectangular patterns on top of traffic signs to cause misclassification. Very recently, localized adversarial attacks, i.e., Adversarial patch  and LaVAN  have been introduced that can be optimized for triplets (misclassification confidence, target class, perturbed location). These practical attacks have demonstrated high strength and can easily bypass existing defense approaches. Therefore they present a significant challenge for existing deep learning systems.
Contributions: In this work, we study the behavior of localized adversarial attacks and propose an effective mechanism to defend against them (see Fig. 1). LaVAN and Adversarial patch add adversarial noise without affecting the original object in the image, and to some extent, they are complementary to each other. In an effort towards a strong defense against these attacks, this paper contributes as follows:
Motivated by the observation that localized adversarial attacks introduce high-frequency noise, we propose a transformation called Local Gradient Smoothing (LGS). LGS first estimates region of interest in an image with the highest probability of adversarial noise and then performs gradient smoothing in only those regions.
We show that by its design, LGS significantly reduces gradient activity in the targeted attack region and thereby showing the most resistance to BPDA , an attack specifically designed to bypass transformation based defense mechanisms.
2 Related Work
Among the recent localized adversarial attacks, the focus of adversarial patch  is to create a scene independent physical-world attack that is agnostic to camera angles, lighting conditions and even the type of classifier. The result is an image independent universal noise patch that can be printed and placed in the classifier’s field of view in a white box (when deep network model is known) or black box (when deep network model is unknown) setting. However, the size of the adversarial patch should be 10% of the image for the attack to be successful in about 90% cases . This limitation was addressed by Karmoon , who focused on creating localized attack covering as little as 2% of the image area instead of generating a universal noise patch. In both of these attacks [4, 12], there is no constraint on noise, and it can take any value within image domain, i.e., [0, 255] or [0, 1].
Defense mechanisms against adversarial attacks can be divided into two main categories. (a) Methods that modify DNN by using adversarial training  or gradient masking  and (b) techniques that modify input sample by using some smoothing function to reduce adversarial effect without changing the DNN [6, 5, 9, 26]. For example, JPEG compression was first presented as a defense by  and recently studied extensively by [5, 19].  presented feature squeezing methods including bit depth reduction, median filtering, Gaussian filtering to detect and defend against adversarial attacks. Guo  considered smoothing input samples by total variance minimization along with JPEG compression and image quilting to reduce the adversarial effect. Our work falls into the second category as we also transform the input sample to defend against localized adversarial attacks. However, as we will demonstrate through experiments, the proposed defense mechanism provides better defense against localized attacks compared to previous techniques.
The paper is organized as follows: Section 3 discusses localized adversarial attacks, LaVAN and Adversarial patch in detail. Section 4 presents our defense approach (LGS) against these attacks. We discuss other related defense methods in Section 5.2. Section 5 demonstrates the effectiveness of the proposed method LGS in comparison to other defense methods against LaVAN and adversarial patch attacks. Section 5.3 discusses BPDA and resilience of different defense methods against it. Section 6 concludes the draft by discussing possible future directions.
3 Adversarial Attacks
3.1 Traditional Attacks
The search for adversarial examples can be formulated as a constrained optimization problem. Given a discriminative classifier , an input sample , a target class and a perturbation budget , an attacker seeks to find a modified input with adversarial noise to increase likelihood of the target class by solving the following optimization problem:
This formulation produces well camouflaged adversarial examples but changes each pixel in the image. Defense methods such as JPEG compression [6, 5], Total variance minimization  and Feature squeezing  are effective against such attacks especially when the perturbation budget is not too high.
LaVAN  differs from the formulation presented in Eq. 3.1 as it confines adversarial noise to a small region, usually away from the salient object in an image. It uses the following spatial mask to replace the small area with noise, as opposed to noise addition performed in traditional attacks:
where and represents adversarial noise.
They also introduce a new objective function where at each iteration, optimization algorithm takes a step away from the source class and towards the target class simultaneously, as follows:
where is given by Eq. 2.
3.3 Adversarial Patch
Adversarial examples created using the methodology presented in Eq. 3.1 cannot be used in physical world attacks because adversarial noise loses its effect under different camera angles, rotations and lighting conditions. Athalye  introduced an Expectation over Transformation (EoT) attack to create robust adversarial examples invariant to chosen set of transformations. Brown  build upon Athalye’s work and used EoT to create a scene independent robust noise patch confined to small region that can be printed and placed in the classifier’s field of view to cause misclassification. To generate adversarial patch ,  proposed a patch operator for a given image , patch , location and a set of transformation . During optimization, patch operator applies a set of transformations to the patch and then projects it onto the image at a location to increase likelihood of target class .
where represent training images, represents distribution over transformations, and is a distribution over locations in the image.
4 Defense: Local Gradients Smoothing
Both of the above discussed attacks [12, 4] introduce high frequency noise concentrated at a particular image location and strength of such a noise becomes very prominent in image gradient domain. We propose that the effect of such adversarial noise can be reduced significantly by suppressing high frequency regions without effecting the low frequency image areas that are important for classification. An efficient way to achieve this is by projecting scaled normalized gradient magnitude map onto the image to directly suppress high activation regions. To this end, we first compute the magnitude of first-order local image gradients as follows:
where denote the horizontal and vertical directions in the image plane. The range of the gradient magnitude calculated using the above equation is normalized for consistency across an image as follows:
The normalized gradient is projected onto the original image to suppress noisy perturbations in the input data domain. This operation smooths out very high frequency image details. As demonstrated by our evaluations, these regions have high likelihood of being perturbed areas, but they do not provide significant information for final classification. The noise suppression is performed as follows:
where is the smoothing factor for LGS and is clipped between 0 and 1. Applying this operation at a global image level, however, introduces image structural loss that causes a drop in classifier’s accuracy on benign examples. To minimize this effect, we design a block-wise approach where gradient intensity is evaluated within a local window. To this end, we first divide the gradient magnitude map into a total of overlapping blocks of same size () and then filter these blocks based on a threshold () to estimate highest activation regions which also have the highest likelihood of adversarial noise. This step can be represented as follows:
where denotes the cardinality of each patch, denotes the patch overlap, represent the windowing operation, denote the vertical and horizontal components of the top left corner of the extracted window, respectively. We set the block size with overlap and threshold is in all of our experiments. The updated gradient blocks represented as are then collated to recreate the full gradient image: . Figure 2 shows the effect of windowing search on gradients magnitude maps. We further demonstrated LGS efficiency on challenging images in supplementary material.
|No Attack||42x42 noise patch covering 2% of image||52x52 noise patch covering 3% of image||60x60 noise patch covering 4% of image|
5.1 Protocol and Results Overview
We used Inception v3 model  to experiment with various attack and defense mechanisms in all of our experiments. All attacks are carried out in white-box settings. We consider the validation set available with Imagenet-2012 dataset in our experiments. This set consists of a total of 50k images. We report top-1 accuracy of classifier. Results are summarized in tables 1, 2 and 3.
LaVAN  can be optimized for triplets (target, confidence, location) but it is highly sensitive to noise location. Adversary loses its effect with even a small change to the pixel location. To reduce the computational burden and conduct experiments on a large scale, we randomly chose noise location along border areas of the image because they have the least probability to cover the salient object. We ran 1000 iterations of attack optimization per image. We terminate the optimization early if classifier mis-classify with confidence above than or equal to 99% or we let it run for at max 1000 iterations and attack is considered to be successful if the image label is changed to a random target (not equal to the true object class). Inceptionv3 model accepts 299x299 image as an input. Three adversarial noise masks with size 42x42 (2% of the image), 52x52 (3% of the image) and 60x60 (4% of the image) were applied. Table 1 presents summary of all the results. For the case of adversarial patch  attack, placing a patch of size 95x95 ( 10% of the image) randomly on all Imagenet validation set was not possible because it would cover most of salient objects details in an image. So we carefully created 1000 adversarial examples that model misclassified as a toaster with a confidence score at least 90%. We then applied all the defense techniques and reported results in Table 2. Figure 3 shows runtime of defense methods to process ImageNet  validation set. We used optimized python implementations. Specifically, we employed JPEG from Pillow, Total variance minimization (TVM), and Bilateral filtering (BF) from scikit-image, Median filtering (MF) and Gaussian filtering (GF) from scipy, and LGS and Bit Depth Reduction (BR) are written in python 3.6 as well. All experiments were conducted on desktop windows computer equipped with Intel i7-7700k quad-core CPU clocked at 4.20GHz and 32GB RAM.
, DW in blind defense scenario, MF with window equal to 3, JPEG compression with quality equal to 30, TVM with weights equal to 10 and BR with depth 3. This hyperparameter choice was made for fair comparison such that the performance on benign examples from ImageNet is approximately the same (first column of Table1). Results are reported for 1000 adversarial examples misclassified as toaster with confidence above than 90%.
5.2 Comparison with Related Defenses
In this section, we report comparisons of our approach with other recent defense methods that transform the input sample to successfully reduce the adversarial effect. The compared methods include both global and local techniques. Note that our method processes image locally so it has advantage over other defenses like JPEG, MF, TVM and BR that process image globally. First, we provide a brief description of the competing defenses which will allow us to elaborate further on the performance trends in Tables 1, 2 and 3.
5.2.1 Digital Watermarking
Hayes et.al  presented two, non-blind and blind, defense strategies to tackle the challenge of localized attacks [12, 4]. Non-blind defense considers a scenario, where defender has the knowledge of adversarial mask location. This is unlikely scenario in the context of adversarial attacks because threat is over immediately, once the adversary provides the mask location. Localized attacks have the ability to change the attention of classifier from the original object to adversarial mask. In their blind defense, authors  exploited the attention mechanism by first finding the mask location using saliency map and then processing that area before inference. Using saliency map to detect adversarial mask location is the strength of this defense but at the same time its also the weakness of defense because on benign examples, saliency map will give the location of original object and hence processing original object will decrease the performance on clean examples. Authors  reported blind defense performance to protect VGG19  on only 400 randomly selected images with 12% accuracy drop on clean images. We have tested this defense on imagenet validation set  (50k images). This method has the second best accuracy on adversarial examples after LGS but its accuracy on clean examples expectedly dropped by a large margin (22.8%). Tables 1, 2 and 3 summarizes the performance of digital watermarking .
5.2.2 JPEG Compression
[6, 5, 19] extensively studied JPEG compression to defend against adversarial attacks. This way high-frequency components are removed that are less important to human vision by using Discrete Cosine Transform (DCT). JPEG performs compression as follows:
Convert an RGB image color space, where and , represent luminance and chrominance respectively.
Down-sample the chrominance channels and apply DCT to blocks for each channel.
Perform quantization of frequency amplitudes by dividing with a constant and rounding off to the nearest integer.
As illustrated in Table 1, image quality decreases as the degree of compression increases which in turn decreases accuracy on benign examples. JPEG compression is not very effective against localized attacks, and its defending ability decreases a lot against BPDA. JPEG performance comparison is shown in Tables 1, 2 and 3 and Figure 4.
5.2.3 Feature Squeezing
The main idea of feature squeezing  is to limit the explorable adversarial space by reducing resolution either by using bit depth reduction or smoothing filters. We found that bit reduction is not effective against localized attacks, however smoothing filter including Gaussian filter, median filter, and bilateral filter reduces localized adversarial effect with reasonable accuracy drop on benign examples. Among smoothing filters, median filter outperforms Gaussian and bilateral filters. Feature squeezing performance is shown in Tables 1, 2 and 3 and Figure 4.
5.2.4 Total Variance Minimization (TVM)
Guo  considered smoothing adversarial images using TVM along with JPEG compression and image quilting. TVM has the ability to measure small variations in the image, and hence it proved effective in removing small perturbations. As illustrated in Table 1, TVM becomes ineffective against large concentrated variations introduced by the localized attacks. Further comparisons are shown in Tables 2 and 3 and Figure 4.
5.3 Resilience to BPDA
BPDA  is built on the intuition that transformed images by JPEG or TVM should look similar to original images, that is, . BPDA approximate gradients for non-differentiable operators with combined forward propagation through operator and DNN while ignoring operator during the backward pass. This strategy allows BPDA to approximate true gradients and thus bypassing the defense. In the traditional attack setting like Projected Gradient Descent (PGD) , the explorable space available to BPDA is because it can change each pixel in the image. In localized attack setting explorable space reduces to controlled by the mask size. LGS suppresses the high-frequency noise to near zero thereby significantly reducing gradient activity in the estimated mask area and restricting BPDA to bypass defense. However, as it is the case with all defenses, increasing explorable space, i.e., distance limit in PGD attack  and mask size in the case of localized attack , protection ability of defense methods decreases. To test performance against BPDA in the localized setting, we randomly selected 1000 examples from Imagenet and attack is optimized against all defenses for the same target, location, mask size and number of iterations. Compared to other defenses methods, LGS significantly reduces the explorable space for localized adversarial attacks within mask size equal to of the image as discussed in . In the case of DW  defense, we tested BPDA against the proposed input processing given the mask location. Summary of attack success rate against defense methods is presented in Table 3.
|LaVAN with BPDA||88%||18%||25.6%||75%||73.30%||78.10%||83%|
6 Discussion and Conclusion
In this work, we developed a defense against localized adversarial attacks by studying attack properties in gradient domain. Defending against continuously evolving adversarial attacks has proven to be very difficult especially with standalone defenses. We believe that in critical security applications, a classifier should be replaced by a robust classification system with following main decision stages:
Detection: given the unlimited distortion space, any image can be converted into an adversarial example that can bypass any defense system with 100% success rate ; however, this also pushes the adversarial example away from the data manifold, and it would be easier to detect rather than removing the perturbation.
Projection or Transformation: Adversarial examples within a small distance of original images can be either projected onto the data manifold or transformed to mitigate the adversarial effect.
Classification: Final stage should be to perform a forward pass through a DNN, whose robustness is increased via adversarial training.
Our method performs a transformation, so it falls into the second stage of robust classification systems. LGS outperforms digital watermarking, JPEG compression, feature squeezing and TVM against localized adversarial attacks with minimal drop in accuracy on benign examples. LGS can be used with a combination of other defense methods, for example, smoothing filters like low pass filter can be applied just on the estimated noisy region to enhance protection for a DNN further.
N. Akhtar and A. S. Mian.
Threat of adversarial attacks on deep learning in computer vision: A survey.IEEE Access, 6:14410–14430, 2018.
-  A. Athalye, N. Carlini, and D. A. Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International Conference on Machine Learning (ICML), 2018.
-  A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok. Synthesizing robust adversarial examples. In International Conference on Machine Learning (ICML), 2017.
-  T. B. Brown, D. Mané, A. Roy, M. Abadi, and J. Gilmer. Adversarial patch. In Neural Information Processing Systems (NIPS), 2017.
-  N. Das, M. Shanbhogue, S.-T. Chen, F. Hohman, S. Li, L. Chen, M. E. Kounavis, and D. H. Chau. Shield: Fast, practical defense and vaccination for deep learning using jpeg compression. In Knowledge Discovery and Data Mining (KDD), 2018.
-  G. K. Dziugaite, Z. Ghahramani, and D. M. Roy. A study of the effect of jpg compression on adversarial images. In International Society for Bayesian Analysis (ISBA), 2016.
I. Evtimov, K. Eykholt, E. Fernandes, T. Kohno, B. Li, A. Prakash, A. Rahmati,
and D. Song.
Robust physical-world attacks on machine learning models.
Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
-  I. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICRL), 2015.
-  C. Guo, M. Rana, M. Cissé, and L. van der Maaten. Countering adversarial images using input transformations. In International Conference on Learning Representations (ICRL), 2017.
-  J. Hayes. On visible adversarial perturbations & digital watermarking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1597–1604, 2018.
-  B. Huval, T. Wang, S. Tandon, J. Kiske, W. Song, J. Pazhayampallil, M. Andriluka, P. Rajpurkar, T. Migimatsu, R. Cheng-Yue, et al. An empirical evaluation of deep learning on highway driving. arXiv preprint arXiv:1504.01716, 2015.
-  D. Karmon, D. Zoran, and Y. Goldberg. Lavan: Localized and visible adversarial noise. In International Conference on Machine Learning (ICML), 2018.
-  A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICRL), 2017.
-  S. M. Moosavi Dezfooli, A. Fawzi, and P. Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), number EPFL-CONF-218057, 2016.
-  N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In Security and Privacy (SP), 2016 IEEE Symposium on, pages 582–597. IEEE, 2016.
-  O. M. Parkhi, A. Vedaldi, A. Zisserman, et al. Deep face recognition. In British Machine Vision Conference(BMVC), volume 1, page 6, 2015.
-  R. Ronen, M. Radu, C. Feuerstein, E. Yom-Tov, and M. Ahmadi. Microsoft malware classification challenge. arXiv preprint arXiv:1802.10135, 2018.
-  O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–252, 2015.
-  U. Shaham, J. Garritano, Y. Yamada, E. Weinberger, A. Cloninger, X. Cheng, K. Stanton, and Y. Kluger. Defending against adversarial images using basis functions transformations. arXiv preprint arXiv:1803.10840, 2018.
-  M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 1528–1540. ACM, 2016.
-  K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition, 2014.
-  J. Su, D. V. Vargas, and S. Kouichi. One pixel attack for fooling deep neural networks. arXiv preprint arXiv:1710.08864, 2017.
-  C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2818–2826, 2016.
-  C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICRL), 2014.
-  F. Tramèr, A. Kurakin, N. Papernot, D. Boneh, and P. McDaniel. Ensemble adversarial training: Attacks and defenses. In International Conference on Learning Representations (ICRL), 2018.
-  W. Xu, D. Evans, and Y. Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv preprint arXiv:1704.01155, 2017.