In recent years, Convolutional Neural Networks (CNN) have greatly advanced performance in various vision tasks, including image recognition[11, 14, 26], object detection [25, 7], and semantic segmentation [4, 19], etc. However, it has been observed [31, 9]
that adding human imperceptible perturbations to input image can cause CNN to make incorrect predictions even if the original image can be correctly classified. These intentionally generated images are usually called adversarial examples[9, 15, 31].
A recent development  has demonstrated that adversarial examples generated from certain models (especially for those learned by iteration-based methods [15, 6]) are less transferable to other models. In other words, those adversarial examples easily overfit to a specific network and achieve much lower attack rate in black-box settings (i.e., attackers have no knowledge of the models they may attack). To remedy this, an ensemble  of multiple networks has been suggested to improve the transferability.
However, ensemble-based attacks suffer from expensive computational overhead, making it difficult to efficiently learn transferable adversarial examples. First, in order to acquire good (i.e., low test error) and diverse (i.e., converge at different local minima) models, people usually independently train them from scratch. To leverage their complementarity, existing methods adopt an intensive aggregation way to fuse the outputs of those networks (e.g
., logits). Consequently, attacking methods in ensemble at most 10 networks restricted by the computational complexity.
In this work, we propose a highly efficient alternative called Ghost Networks to address this issue. The basic principle is to generate a huge number of virtual models built on a network trained from scratch (a base network or a base model). The word “virtual” means that those ghost networks are not stored or trained, which incurs extra time and space cost. Instead, they are simply generated by imposing erosion on certain intermediate structures of the base network and then used on-the-fly. In this case, with an increasing number of models we have, it is apparent that a standard ensemble  would be problematic owing to its complexity. Accordingly, we propose Longitudinal Ensemble, a specialized fusion method for ghost networks, which conducts an implicit ensemble during attack iterations. Consequently, transferable adversarial examples can be easily generated without sacrificing computational efficiency.
To summarize, the contributions of our work can be divided into three folds: 1) Our work is the first one to explore network erosion to learn transferable adversarial examples, which does not solely rely on multi-network ensemble. 2) We observe that the number of different networks actually used for ensemble (intrinsic networks) is essential for the transferability. However, it is less necessary to train different models independently. Instead, ghost networks can be a competitive alternative with extremely low complexity. 3) It is generic. Though appearing to be an ensemble-based method for multi-model attacks, it can be also applied to single-model attacks where only one trained model is accessible. Furthermore, it is also compatible with different network structures, attack methods, and adversarial settings.
Extensive experimental results demonstrate that our work is a computationally cheap plug-in which improves the transferability of adversarial examples. In particular, by re-producing NIPS 2017 adversarial competition , our work outperforms the No.1 attack submission by a large margin, which demonstrates its effectiveness and efficiency.
2 Related Work
The transferability of adversarial examples refers to the property of the same input image easily misclassified by different models. This was first investigated by Szegedy et al.  on MNIST, which later led to the development of black-box attack. Afterward, Liu et al. 
proposed ensemble-based approaches which demonstrate transferability on large-scale datasets like ImageNet.
Optimization-based methods (e.g., Carlini-Wagner attack ) and iteration-based methods (e.g., I-FGSM ) tend to overfit a specific network structure, and thus leading to weak transferability . On the other hand, single-step gradient-based methods, such as Fast Gradient Sign Method (FGSM) , are good at learning stronger transferable adversarial examples but are less successful for the white-box attacks. Taking advantages of  and , Dong et al.  proposed a momentum iterative method to generate adversarial examples with stronger transferability. In order to further avoid overfitting specific models, it uses an ensemble of trained-from-scratch models.
However, how to efficiently learn transferable adversarial examples remains a challenging task. Some works [1, 24, 35] suggest re-training neural networks, e.g., generative adversarial learning , can achieve high transferability. Moreover, Papernot et al.  proposed a query-based method to improve black-box attack performance. However, this requires massive information from the target model. In conclusion, acquiring and integrating information from various models to approximate the target model is the key to achieving better transferability. However, most existing works are inefficient and inadequate to learn adversarial examples with strong transferability. Our work addresses this issue with high efficiency.
3 Ghost Networks
The goal of this work is to learn adversarial examples, with a particular attention on the transferability. Given a clean image , we want to find an adversarial example , which is still visually similar to after adding adversarial noise but fools the classifier. In order to improve the transferability of , we choose to simultaneously attack multiple models. However, unlike existing works [6, 18], we propose Ghost Networks, an highly efficient algorithm that can do both generation and combination an ensemble of diverse models to learn transferable adversarial examples.
3.1 Dropout Erosion
Revisiting Dropout. Dropout 
has been one of the most popular techniques in deep learning. By randomly dropping out units from the model during the training phase, dropout can effectively prevent deep neural networks from overfitting. Some recent works[26, 28, 30, 29] achieve state-of-the-art performances on benchmark datasets by applying dropout to a layer of high-level features.
Let be the activation in the -th layer, at training time, the output after the dropout layer can be mathematically defined as
where denotes an element-wise product andof being . At test time, units in are always present, thus to keep the output is the same as the expected output at training time, is set to be .
Perturbing Dropout. Dropout provides an efficient way of approximately combining different neural network architectures and thereby prevents overfitting. Inspired by this, we propose to generate ghost networks by inserting the dropout layer. In order to make these ghost networks as diverse as possible, we densely apply dropout to every block throughout the base network. Therefore, diversity is not limited to high-level features but applies to all feature levels.
Let be the function between the -th and -th layer, i.e., , then the output after applying dropout erosion is
where , and has the same meaning as in Eq. (1), indicating the probability that is preserved. To keep the expected input of consistency after erosion, the activation of should be divided by .
During the inference, the output feature after -th dropout layer () is
where denotes the composite function, more specifically, .
By combining Eq. (2) and Eq. (3), we observe that when (means ), all elements in are equal to . In this case, we do not impose any perturbations to the base network. When gradually increases to (meaning decreases to ), the ratio of the number of elements that are dropped out is . In other words, of elements can be successfully back-propagated. Hence, larger implies a heavier erosion on the base network. Therefore, we define to be the magnitude of erosion.
When perturbing the dropout layer, the gradient in back-propagation can be written as
Generating Ghost Network. The generation of ghost networks via perturbing dropout layer proceeds in three steps: 1) randomly sample a parameter set from the Bernoulli distribution ; 2) apply Eq. (2) to the base network with the parameter set and get the perturbed network; 3) we independently sample for times and obtain a pool of ghost networks which can be used for adversarial attacks.
3.2 Skip Connection Erosion
The standard residual block in  is defined by
where and are the input and output to the -th residual block with the weights . denotes the residual function. As suggested in , it is crucial to uses the identity skip connection, i.e., , to facilitate the residual learning process, otherwise the network may not converge to a good local minima.
Perturbing Skip Connection. Following the principle of skip connection, we propose to perturb skip connection to generate ghost networks.
More specifically, the network weights are first learned using identity skip connections, then switched to the randomized skip connection (see Fig. 2). To achieve this end, we apply randomized modulating scalar to the -th residual block, by
is drawn from the uniform distribution. One may have observed several similar formulations on skip connection to improve the classification performance, e.g., gated inference in ConvNet-AIG  and lesion study in . However, our work focuses on attacking the model with a randomized perturbation on skip connection, i.e., the model is actually not trained via Eq. (6).
During inference, the output feature after -th layer () is
The gradient in back-propagation is then written as
Generating Ghost Network. The generation of ghost networks via perturbing the skip connections is similar to the generating procedure via perturbing the dropout layer. The only difference is that in the first step, we need to sample a set of modulating scalars from the uniform distribution for each skip connection.
3.3 Longitudinal Ensemble
Existing iteration-based ensemble-attack approaches [6, 18] require averaging the outputs (e.g., logits, classification probabilities, losses) of different networks. However, such a standard ensemble  is too costly and inefficient for us because we can readily obtain a huge candidate pool of qualified neural models by using Ghost Networks.
To remedy this, we propose longitudinal ensemble, a specialized fusion method for Ghost Networks, which constructs an implicit ensemble of the ghost networks by randomizing the perturbations during the iterations of adversarial attack (e.g., I-FGSM  and MI-FGSM ). Suppose we have a base model , which can then generate a pool of networks , where is the model number. The key step of longitudinal ensemble is that at the -th iteration, we attack the ghost network only. In comparison, for each iteration, the standard ensemble requires fusing the gradients of all the models in the model pool , which requires more computational cost. We illustrate the difference between the standard ensemble and longitudinal ensemble method in Fig. 3.
The longitudinal ensemble shares the same prior as [6, 18] that if an adversarial example is generated by attacking multiple networks, then it is more likely to transfer to other networks. However, the longitudinal ensemble removes duplicated computations by only sampling one model from the model pool rather than using all models in each iteration.
Three noteworthy comments should be made here. First, the ghost networks are never stored or trained. As a result, this incurs neither additional time nor space complexity. Second, it is obvious from Fig. 3 that attackers can combine the standard ensemble and the longitudinal ensemble of ghost networks. Finally, it is easy to extend longitudinal ensemble to multi-model attack by considering each base model as a branch (see experimental evaluations for details).
In this section, we give a comprehensive experimental evaluation of the proposed Ghost Networks. In order to distinguish models trained from scratch and the ghost networks we generate, we call the former one the base network or base model in the rest of this paper.
Due to space limitations, we will give a more detailed evaluation in the supplementary material.
4.1 Experimental Setup
Base Networks. base models are used in our experiments, including normally trained models111Available at https://github.com/tensorflow/models/tree/master/research/slim, i.e., Resnet v2-50 (Res-50) , Resnet v2-101 (Res-101) , Resnet v2-152 (Res-152) , Inception v3 (Inc-v3) , Inception v4 (Inc-v4)  and Inception Resnet v2 (IncRes-v2) , and adversarially-trained models 222Available at https://github.com/tensorflow/models/tree/master/research/adv_imagenet_models, i.e., Inc-v3ens3, Inc-v3ens4 and IncRes-v2ens.
Datasets. Because it is less meaningful to attack images that are originally misclassified, we select images from the ILSVRC 2012 validation set following , which can be correctly classified by all the base models.
Attacking Methods. We employ two iteration-based attack methods to evaluate the adversarial robustness, i.e., Iterative Fast Gradient Sign Method (I-FGSM) and Momentum Iterative Fast Gradient Sign Method (MI-FGSM). Both of them are variants of Fast Gradient Sign Method (FSGM) , and are available at cleverhans library .
I-FGSM was proposed by Kurakin et al. , and learns the adversarial example by
is the loss function of a network with parameter. is the clip function which ensures the generated adversarial example is within the -ball of the original image with ground-truth label . is the iteration number and is the step size. MI-FGSM was proposed by Dong et al. , and integrates the momentum term into the attack process to stabilize the update directions and escape from poor local maxima. At the -th iteration, the accumulated gradient is calculated by:
where is the decay factor of the momentum term. The sign of the accumulated gradient is then used to generate the adversarial example, by
4.2 Analysis of Ghost Networks
As analyzed above, in order to generate adversarial examples with good transferability, there are generally two requirements for the intrinsic models. First, each individual model should have a low test error. Second, different models should be as diverse as possible (i.e., converge at different local minima). To show the generated ghost networks are qualified for adversarial attack, we conduct an experiment on the whole ILSVRC 2012 validation set .
Descriptive Capacity. In order to quantitatively measure the descriptive capacity of the generated ghost networks, we plot the relationship between the magnitude of erosion and top-1 classification accuracy.
We apply dropout erosion in Sec. 3.1 to non-residual networks (Inc-v3 and Inc-v4) and skip connection erosion in Sec. 3.2 to residual networks (Res-50, Res-101, Res-152 and IncRes-v2). Fig. 4 and Fig. 4 present the change curves of the dropout erosion and skip connection erosion, respectively.
It is not surprising to observe that the classification accurayies of different models are negatively correlated to the magnitude of erosion . By choosing the performance drop approximately equal to 10% as a threshold, we can determine the value of individually for each network. Although the performances of the ghost networks are slightly worse than those achieved by the independently trained base networks, the ghost networks still preserve low error rates. As emphasized throughout this paper, it is extremely cheap to generate a huge number of ghost networks.
In our following experiments, we set to , , , , and for Inc-v3, Inc-v4, Res-50, Res-101, Res-152 and IncRes-v2, respectively unless otherwise specified.
Model Diversity. To measure the diversity, we use Res-50 as the backbone model. We denote the base Res-50 described in Sec. 4.1 as Res-50-A, and independently train two additional models with the same architecture, denoted by Res-50-B and Res-50-C. Meanwhile, we apply skip connection erosion to Res-50-A, then obtain three ghost networks denoted as Res-50S-A, Res-50S-B and Res-50S-C, respectively.
., the predictions after softmax layer) for each pair of networks as in. Given any image, let and denote the softmax outputs of two networks, then JSD is defined as
where is the average of and , i.e., .
is the Kullback-Leibler divergence.
In Fig. 5, we report the averaged JSD for any pairs of networks over the ILSVRC 2012 validation set. As can be drawn, the diversity between ghost networks is comparable or even larger than independently trained networks.
Based on the analysis of descriptive capacity and model diversity, we can see that the generated ghost networks can provide accurate enough yet diverse descriptions of the data manifold, which is beneficial to learn transferable adversarial examples as we will experimentally prove below.
4.3 Single-model Attack
|I-FGSM ||Exp. S1||99.5||16.3||99.4||17.8||98.4||16.7||94.8||8.3||99.8||5.3||99.5||7.3|
|Exp. S3 (ours)||99.7||23.4||99.6||23.7||99.4||21.1||96.5||11.2||97.0||6.3||86.8||10.0|
|Exp. S5 (ours)||99.6||35.9||99.7||35.9||99.6||60.1||98.7||14.6||99.9||12.3||98.5||19.4|
|MI-FGSM ||Exp. S1||99.4||29.4||99.2||31.3||98.3||29.6||94.0||20.0||99.8||13.7||99.5||18.4|
|Exp. S3 (ours)||99.7||39.4||99.8||40.1||99.5||38.0||95.9||26.8||98.0||17.6||90.6||22.4|
|Exp. S5 (ours)||99.6||50.6||99.7||51.4||98.6||64.9||98.3||33.3||99.8||28.3||97.8||37.4|
Firstly, we evaluate the ghost networks in single-model attack, where attackers can only access one base model trained from real data. To demonstrate the effectiveness of our method, we design five experimental comparisons, as
Exp. S2: We apply erosion to the base model and obtain one ghost network . Then, an adversarial attack is conducted to to generate the adversarial examples.
Exp. S3: We independently apply erosion times to get a pool of ghost networks , then utilize the proposed longitudinal ensemble to efficiently fuse them during adversarial attack.
Exp. S4: Similar to Exp. S3, the only difference is that we use the standard ensemble method proposed in  to fuse the ghost networks.
Exp. S5: ghost networks are generated, which are fused in a manner, that is, we do a standard ensemble of models for each iteration of attack and a longitudinal ensemble of models.
We attack normally-trained networks, and test on all the models ( adversarially-trained network are included). The attack rate is shown in Table 1. Due to the space limitation, we report the average performances for black-box attack, rather than each individual performance on each testing model (all the individual cases are shown in Fig. 6 and the supplementary material.).
As can be drawn from Table 1, a single ghost network is worse than the base network (Exp. S2 vs. Exp. S1), due to the fact that the descriptive power of ghost network is inferior to base network. However, by leveraging the longitudinal ensemble, our work achieves a much higher attack rate at most settings, especially for the black-box atack. For example, when attacking Res-50 in black-box attack, Exp. S3 outperforms Exp. S1 by with I-FGSM and by with MI-FGSM. This observation firmly demonstrates the effectiveness of ghost networks in learning transferable adversarial examples. It should be mentioned that the computational cost almost remains the same as Exp. S1 for two reasons. First, the ghost networks used in Exp. S3 are not trained but eroded from the base model and used on-the-fly. Second, multiple ghost networks are fused via the longitudinal ensemble, instead of the standard ensemble method in .
In fact, the proposed ghost networks can be also fused via the standard ensemble method, as shown in Exp. S4. In this case, we can report a higher attack rate at the sacrifice of computational efficiency. For instance, Exp. S4 reports attack rate of by attacking Res-50 with I-FGSM in black-box setting, an improvement of over Exp. S3.
This observation, from another point of view, inspires us to combine the standard ensemble and the longitudinal ensemble as shown in Exp. S5. As we can see, Exp. S5 consistently beats all the compared methods in all the black-box settings. Of course, Exp. S5 is as computational expensive as Exp. S4. However, the additional computational overhead stems from the standard ensemble, rather than longitudinal ensemble proposed in this work.
Note that in all the experiments presented in Table 1, we use only one individual base model. Even in the case of Exp. S3, all the to-be-fused models are ghost networks. However, the generated ghost networks are never stored or trained, which means no extra space complexity are needed. Therefore, one can clearly observe the benefit of ghost networks. Especially when comparing Exp. S5 and Exp. S1, ghost networks can achieve a dramatic improvement in black-box attack.
Based on the experimental results above, we arrive a similar conclusion as , that is, the number of intrinsic models is essential to improve the transferability of adversarial examples. However, a different conclusion is that it is less necessary to independently train different models. Instead, ghost networks can be a computationally cheap alternative which can enable a good performance. When the number of intrinsic models increases, the attack rate will increase. We will further exploit this in multi-model attack.
In Fig. 6, we select two base models, i.e., Res-50 and Inc-v3, to attack and present their individual performances when testing on all the base models. One can easily observe the positive effect of ghost networks on improving the transferability of adversarial examples.
4.4 Multi-model Attack
In this subsection, we evaluate ghost networks in multi-model setting, where attackers have access to multiple networks trained independently.
4.4.1 Same Architecture and Different Parameters
We first evaluate a simple setting of multi-model attack, where the base models share the same network architecture but have different weights. The same three Res-50 models as in Sec. 4.2 are used, denoted as , and . Then, we denote the -th ghost network generated upon as .
Exp. M1: A standard ensemble of the base model for three times. This is simply equivalent to single-model attack, which can serve as a weak baseline.
Exp. M2: A standard ensemble of the base models and and , which can serve as a strong baseline.
Exp. M3: A standard ensemble of and and , which simply replaces the base model in Exp. 1 with three ghost networks associated to it.
Exp. M4: A standard ensemble of and and , which replaces the base networks used in Exp. 2 with ghost networks, each one associated to a base model.
Exp. M5: ghost networks are generated upon the base model . They are fused in a manner, that is, we do a standard ensemble of models for each iteration of attack, and a longitudinal ensemble of models in total.
Exp. M6: For each base model, we generate ghost networks. At the -th iteration of attack, we do a standard ensemble of , then do a longitudinal ensemble of for the -th base model.
The adversarial example generated by each method are used to test all the models. We report the average attack rates in Table 2. It is easy to understand that Exp. M2 performs better than Exp. M1, Exp. M3 and Exp. M4 as it has three independently trained models. However, by comparing Exp. M5 with Exp. M2, we observe a significant improvement of attack rate. For example, By using MI-FGSM as the attack method, Exp. M5 beats Exp. M2 by . Although Exp. M5 only has base model and Exp. M2 has , Exp. M5 actually fuses intrinsic models. Such a result further supports our previous claim that the number of intrinsic models is essential but it is less necessary to independently obtain them by training from scratch. Similarly, Exp. M6 yields the best performance as it has independently trained models and intrinsic models.
|Methods||Attack Rate||Model Number|
|I-FGSM + ours||92.86||93.04||92.62||96.02||95.46||96.82|
|MI-FGSM + ours||93.98||93.88||93.66||96.96||95.92||97.08|
|I-FGSM + ours||80.22||79.80||77.02||60.20||73.18||67.84|
|MI-FGSM + ours||87.14||86.14||84.64||74.18||82.06||79.18|
|I-FGSM + ours||21.38||22.00||21.78||20.98||24.06||21.36|
|MI-FGSM + ours||34.10||34.50||35.00||33.78||39.78||36.64|
|I-FGSM + ours||11.30||11.74||11.56||10.10||12.98||10.98|
|MI-FGSM + ours||17.82||17.68||17.78||16.06||22.16||18.82|
|I-FGSM + ours||18.42||18.26||18.66||17.94||20.08||17.40|
|MI-FGSM + ours||29.32||28.98||29.58||29.00||32.60||30.48|
|Methods||Black-box Attack||White-box Attack|
4.4.2 Different Architectures
Besides the baseline comparison above, we then evaluate ghost networks in the multi-model setting following . We attack an ensemble of out of normally-trained models in this experiment, then test on the ensembled network (white-box setting) and the hold-out network (black-box setting). We also test on the adversarially-trained networks to evaluate the transferability of the generated adversarial examples in black-box attack.
The results are summarized in Table 3, our method achieves comparable attack rates on the ensembled network (white-box setting) than I-FGSM and MI-FGSM. However, the performances in black-box attack are significantly improved. For example, when holding out Res-50, our method improves the performance of I-FGSM from to , and that of MI-FGSM from to . When testing on the three adversarially-trained networks, the improvement is more notable. These results further testify to the ability of ghost networks to learn transferable adversarial examples.
4.5 NIPS 2017 Adversarial Challenge
Finally, we evaluate our method in a benchmark test of the NIPS 2017 Adversarial Challenge . For performance evaluation, we use the top-3 defense submissions (black-box models), i.e., TsAIL333https://github.com/lfz/Guided-Denoise, iyswim444https://github.com/cihangxie/NIPS2017_adv_challenge_defense and Anil Thomas555https://github.com/anlthms/nips-2017/tree/master/mmd, and three official baselines (white-box models), i.e., Inc-v3adv, IncRes-v2ens and Inc-v3. The test dataset contains images with the same 1000-class labels as ImageNet .
Following the experimental setting of the No.1 attack submission , we attack on an ensemble of Inc-v3, IncRes-v2, Inc-v4, Res-152, Inc-v3ens3, Inc-v3ens4, IncRes-v2ens and Inc-v3adv . The ensemble weights are set to / equally for the first seven networks and / for Inc-v3adv. The total iteration number is set to , and the maximum perturbation is randomly selected from . The step size .
The results are summarized in Table 4. Consistent with previous experiments, we observe that by applying ghost networks, the performance of the No. 1 submission can be significantly improved, especially with black-box attack. For example, the average performance of black-box attack is changed from to , an improvement of . The most remarkable improvement is achieved when testing on iyswim, where ghost networks leads to an improvement of . This suggests that our proposed method can generalize well to other defense mechanisms.
This work focuses on learning transferable adversarial examples for adversarial attack. We propose, for the first time, to exploit network erosion to generate a kind of virtual models called ghost networks. Ghost networks, together with the coupled longitudinal ensemble strategy, require almost no additional time and space consumption, therefore can be a rather efficient tool to improve existing methods in learning transferable adversarial examples. Extensive experiments (more in the supplementary material) have firmly demonstrated the efficacy of ghost networks. Note that the ghost networks in our work are generated by perturbing the dropout layer and skip connectection. However, it would be interesting to see the effect if other typical layers (e.g13]
, relu) in neural networks are perturbed. We leave these issues as future work.
S. Baluja and I. Fischer.
Learning to attack: Adversarial transformation networks.In AAAI, 2018.
B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šrndić, P. Laskov,
G. Giacinto, and F. Roli.
Evasion attacks against machine learning at test time.In ECML-PKDD, 2013.
-  N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In IEEE S&P, 2017.
-  L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 40(4):834–848, 2018.
-  J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
-  Y. Dong, F. Liao, T. Pang, H. Su, X. Hu, J. Li, and J. Zhu. Boosting adversarial attacks with momentum. In CVPR, 2018.
-  R. Girshick. Fast r-cnn. In ICCV, 2015.
-  I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NIPS, 2014.
-  I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
-  K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
-  K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In ECCV, 2016.
-  G. Huang, Y. Li, G. Pleiss, Z. Liu, J. E. Hopcroft, and K. Q. Weinberger. Snapshot ensembles: Train 1, get m for free. In ICLR, 2017.
-  S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
-  A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
-  A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial examples in the physical world. In ICLR Workshop, 2017.
-  A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial machine learning at scale. In ICLR, 2017.
-  A. Kurakin, I. Goodfellow, S. Bengio, Y. Dong, F. Liao, M. Liang, T. Pang, J. Zhu, X. Hu, C. Xie, et al. Adversarial attacks and defences competition. arXiv preprint arXiv:1804.00097, 2018.
-  Y. Liu, X. Chen, C. Liu, and D. Song. Delving into transferable adversarial examples and black-box attacks. In ICLR, 2017.
-  J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
-  V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
-  N. Papernot, F. Faghri, N. Carlini, I. Goodfellow, R. Feinman, A. Kurakin, C. Xie, Y. Sharma, T. Brown, A. Roy, A. Matyasko, V. Behzadan, K. Hambardzumyan, Z. Zhang, Y.-L. Juang, Z. Li, R. Sheatsley, A. Garg, J. Uesato, W. Gierke, Y. Dong, D. Berthelot, P. Hendricks, J. Rauber, R. Long, and P. McDaniel. cleverhans v2.1.0: an adversarial machine learning library. arXiv preprint arXiv:1610.00768, 2018.
-  N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami. The limitations of deep learning in adversarial settings. In EuroS&P, 2016.
-  N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In IEEE S&P, 2016.
-  O. Poursaeed, I. Katsman, B. Gao, and S. Belongie. Generative adversarial perturbations. In CVPR, 2017.
-  S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015.
-  K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
-  N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 15(1):1929–1958, 2014.
C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi.
Inception-v4, inception-resnet and the impact of residual connections on learning.In AAAI, 2017.
-  C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR, 2015.
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna.
Rethinking the inception architecture for computer vision.In CVPR, 2016.
-  C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In ICLR, 2014.
-  F. Tramèr, A. Kurakin, N. Papernot, D. Boneh, and P. McDaniel. Ensemble adversarial training: Attacks and defenses. In ICLR, 2018.
-  A. Veit and S. Belongie. Convolutional networks with adaptive inference graphs. In ECCV, 2018.
-  A. Veit, M. J. Wilber, and S. Belongie. Residual networks behave like ensembles of relatively shallow networks. In NIPS, 2016.
-  C. Xiao, B. Li, J.-Y. Zhu, W. He, M. Liu, and D. Song. Generating adversarial examples with adversarial networks. In IJCAI, 2018.
-  C. Xie, Z. Zhang, J. Wang, Y. Zhou, Z. Ren, and A. Yuille. Improving transferability of adversarial examples with input diversity. arXiv preprint arXiv:1803.06978, 2018.