Deep learning is being increasingly adopted within the medical imaging community for a plethora of tasks including classification, segmentation, detection etc. The classic approach towards the assessment of any machine learning model revolves around the evaluation of its generalizability i.e. its performance on unseen test scenarios. However, in case of limited training data, such as medical imaging datasets, using heavily over-parameterized deep learning models could lead to the ”memorization” of the training data. Evaluating such models on an available non-overlapping test set is popular, yet significantly limited in its ability to explore the model’s resilience to outliers and noisy data / labels (i.e. robustness). Additionally, the limited interpretability of deep learning models due to their ”black-box” nature challenges their adoption into clinical practice.
Existing model evaluation routines look deeply into over-fitting but insufficiently into scenarios of model sensitivity to variations of the input. Robustness evaluation estimates potential failure probabilities when the model ispushed to its limits. In this paper, we approach evaluating a model by leveraging adversarial examples  that are crafted with the purpose of fooling a model and can uncover cases where its performance may degenerate. Our approach to using adversarial examples as benchmark is also significantly less laborious and expensive than constituting a sufficiently diverse test set with manual annotation.
Adversarial examples are images crafted to purposely fool machine learning models, while the added perturbations are imperceptible to human eyes , as shown in Fig. 1. To the best of our knowledge, this is the first paper to explore adversarial examples in medical image computing and leverage them in a constructive fashion to benchmark model performance not only on clean and noisy but also on adversarially crafted data. It must be noted that though these examples may not occur in naturally acquired data, utilizing them can present new opportunities for medical imaging researchers to uncover more about their models, with the ultimate goal of increasing robustness and optimizing the decision boundaries learned for different tasks.
Our contribution is two-fold: Firstly, we demonstrate on a variety of medical image computing tasks that widely adopted state-of-the art deep learning models are not immune to adversarial examples crafting. Secondly, we utilize adversarial examples to benchmark model robustness by comparing a variety of architectures, such as Inception  and UNet , for the tasks of skin lesion classification and whole brain segmentation.
2.1 Adversarial Crafting
Given a trained model , an original input image with output label , we generate an adversarial example by solving a box-constrained optimization problem subject to , , and . Such an optimization minimizes the added perturbation, say (i.e. ) while simultaneously fooling the model . By imposing an additional constraint such as , we can restrict the perturbation to be small enough to be imperceptible to humans.
Classification: Gradient-based adversarial example generation methods have been proposed with the objective of generating minimum amount of perturbation
that mis-classifies. These include the Fast Gradient Sign Method (FGSM) , DeepFool (DF) , Saliency Map Attacks (SMA)  etc. Adversarial examples crafted with these methods are shown in Fig. 2. For a trained model , FGSM performs a one-step pixel-level update along the sign of the gradient that maximizes the task loss and the resultant perturbation is computed as , where are the parameters of the model. The amount of perturbation is regulated by a hyper-parameter that is typically assigned a low value, so that is visually imperceptible from .
Differing from FGSM, DF follows an iterative greedy search process, where in each iteration the projections of the input sample to the decision boundaries of all the classes are computed and an is calculated that will push towards the closest decision boundary of a class, other than the correct one. In SMA, the impact of each pixel on the prediction of the model is estimated and the input is selectively perturbed to effect the most significant change to the output.
Segmentation: In , the authors introduced Dense Adversarial Generation (DAG) as a method for crafting adversarial examples for semantic segmentation in a fast and effective way, closely resembling per-pixel, targeted FGSM. Particularly, DAG utilizes an incorrect segmentation mask, given by the user, and a target set of non-background pixels. Its goal is to calculate a minimum perturbation that will alter the prediction on the pixels in the target set from the correct class to the incorrect target class.
Specifically, DAG minimizes the distance between the prediction for pixels of the ground truth and the incorrect target as shown in:
are the logits of the model andthe number of classes. represents a target set of non-background pixels that DAG is allowed to perturb in order to constrain the search-space of the perturbation. In step the image has been transformed to , where the perturbation is computed by:
We utilized DAG to craft adversarial examples, seen in Fig. 2, by creating targets with varying degrees of difficulty. Particularly, we set the target to be all background (Type A), randomly assign a small percentage of pixels to a randomly-selected adversarial class (Type B) and modify (dilate) only a particular target class while keeping all other classes intact (Type C). Of the aforementioned attack types, Type A is the most challenging, causing the largest amount of perturbation, while Type C is expected to distort the image the least, as can be seen in Fig. 2. The Mean Square Error (MSE) between the original and adversarial images remained extremely small, ranging from 0.004 for adversaries of Type A to 0.002 for B and C.
2.2 Model Evaluation With Adversarial Examples
Our approach involves benchmarking models against a number of task-specific adversarial attacks discussed in Sec. 2.1 that are crafted implicitly from the data. The proposed pipeline for the evaluation of robustness is similar across both classification and segmentation. For classification, we crafted adversarial examples with FGSM, DF and SMA, while for segmentation we applied DAG with 3 different types of targets (Type A-C). Afterwards we attacked our models in a black-box fashion with examples generated by independently trained models, to maintain an unbiased attack scenario. In addition to that, black-box attacks highlight the transferability of these examples between a variety of different models .
Contrasting with Noise: One could argue that applying noise on the test images before inference could replace the need for adversarial examples. However, that is not the case since hard ambiguous cases and outliers cannot be modeled by noise distributions. Adversarial examples, which are crafted with the purpose to force models to fail, are better suited for evaluating model behavior when subject to input extrema. To showcase that adversarial perturbations do not resemble noise distributions, we also crafted images distorted with modality-specific noise (Gaussian noise for dermatoscopic images and Rician noise for T1w MRI). For fairness, the Structural Similarity (SSIM) between the original and noisy images was the same as the one between the original and adversarial examples and ranged from 0.97 to 0.99.
We plot the t-Stochastic Neighbor Embedding representation (t-SNE) from IV3 for the clean, noisy and adversarial examples (FGSM) in Fig. 3 for the classification task to further illustrate this difference. Contrasting Fig. 3(L) with Fig. 3(R), we clearly observe that images distorted with noise are embedded close to the clean images, while adversarial examples are pushed further towards other classes. The anomalous nature of the adversarial examples clearly supports our hypothesis that their behavior is not akin to noise and can act as a harder benchmark for evaluating a model’s robustness.
To provide a proof-of-concept for the proposed robustness evaluation we chose the challenging tasks of fine-grained skin lesion classification using dermatoscopic images and segmentation of the whole brain in T1w-MRI scans. The task-specific model learning is described as follows:
Classification: We fine-tune three state-of-the art deep learning architectures namely, InceptionV3 (IV3) , InceptionV4 (IV4)  and MobileNet (MN)  for this task. Both IV3 and IV4 are very-deep architectures (
100 layers), while MN is significantly compact. Comparing these architectures would help discover if any innate relationships exist between model complexity (in terms of depth and parameters) and their robustness. To keep the comparisons fair, all the models were initialized with their respective ImageNet parameters and fine-tuned with a weighted cross-entropy loss with affine data augmentation. Specifically, the models were trained with stochastic gradient descent with a decaying learning rate initialized at 0.01, momentum of 0.9 and dropout of 0.8 for regularization. We use the publicly-available Dermofit image library consisting of 1300 high-quality dermatoscopic images, with histologically validated fine grained expert annotations (10 classes) for this task. The dataset was split at patient-level with non-overlapping folds (50% for training and rest for testing).
Segmentation: For this task we chose to evaluate three popular fully-convolutional deep architectures, namely SegNet (SN) , UNet (UN)  and DenseNet (DN) . Contrasting across these architectures, we evaluate the importance of skip connections with respect to robustness varying from no skip connections in SN to introducing long-range skips in UN and both long and short-range skip connections in DN. The model parameters (depth and layers) were chosen to maintain comparable model complexity, so as to exclusively factor out the impact of skip connections to robustness. The aforementioned models were trained with a composite loss of weighted-cross entropy and Dice loss as proposed in  and model optimization was performed with ADAM optimizer with an initial learning rate of 0.001. We use 27 volumes from the publicly-available whole-brain segmentation benchmark (subset of Open Access Series of Imaging Studies (OASIS) dataset ) that was released as a part of the Multi-Atlas Labeling Challenge in MICCAI 2012 
, with 80-20 patient-level splits for training and testing. The models for both tasks were trained until convergence using the TensorFlow deep learning library and adversarial examples for DF and SMA attacks described in Sec. 2.2 were crafted using the FoolBox  library.
Following the model evaluation strategy presented in Sec. 2.2, adversarial examples were crafted for each of the trained models and their robustness is evaluated in terms of average classification accuracy and average Dice score. We report the overall performance of the models in Table 1, where we compare the performance of each model on clean and noisy images with their average score against all the attacks. Furthermore, in Table 2 we are reporting the performance of each model against all the black-box attacks separately.
4 Results and Discussion
4.1 Robustness Evaluation for Classification
Visual Evaluation: Fig. 2
(Left) illustrates adversarial examples crafted for an unseen test example (belonging to malignant melanoma class) for each of the classification related attacks (FGSM, DF and SMA) alongside an image perturbed with Gaussian noise for comparison. A scaled version of the difference image with respect to the original is also shown along with the posterior probabilities estimated using IV3 network. We observe that all the adversarial examples, regardless of the attack are consistently misclassified with very high confidence, while the addition of Gaussian noise only results in confidence reduction. Furthermore, FGSM induces perturbations dispersed across the whole image, while DF and SMA generate perturbations more localized to the lesion.
Attacks: From Table 1, we observe that IV4 and MN both achieve comparable performance on clean data (80-81%) superior to the one of IV3 (71%). By limiting model evaluation to generalizability (i.e. performance on clean data) one may prematurely conclude that IV3 demonstrates the worst comparative performance. However, upon comparing the robustness of these models with respect to average performance under all the attacks (Table 2), we observe a contrary trend. The performance drop for IV3 is significantly lower (7%) in comparison to IV4 (17%) and MN (25%). IV4 achieves higher accuracy on DF and SMA attacks, while IV3 is the most robust model against FGSM. Contrasting IV4 and MN, we observe that MN performs poorly not only on noisy samples but also on all of the attacks, as shown in Table 2. These contrasting observations clearly substantiate the core hypothesis within the paper that model evaluation should not be limited to generalizability and that performing robustness evaluation is equally important. Despite showing comparable performance in terms of generalizability, we can clearly conclude that IV4 is strongly preferred over MN.
|Type A||Type B||Type C|
4.2 Robustness Evaluation for Segmentation
Visual Evaluation: Fig. 2 (Right) illustrates how the prediction maps of the trained DN model transform when it is attacked by adversarial examples. All the DAG attacks (Type A-C) successfully fool the model into producing an incorrect prediction map. However, the prediction on the image distorted with Rician noise is visually very similar to the one of the original image and the ground truth. This clearly demonstrates that adding adversarial perturbation is not akin to adding randomly generated noise. Thus, adversarial examples are superior in pushing a network to its limits and evaluating its robustness.
Attacks: From Table 1, we observe that DN (86.1%) and UN (86.2%) achieve almost identical performance on clean unseen test examples and fare better than SN (84.2%), highlighting the importance of skip connections. Furthermore, the fact that the performance drop caused by the addition of Rician noise remains low for UN and DN (10% and 1% respectively) reinforces the distinction between noise and adversarial perturbations. Regarding model performance with respect to adversarial attacks in Table 2, we observe that DN is not only resilient to noise but also significantly more robust than SN (by 18%) and UN (by 21%) to attacks crafted by any other model. Furthermore, SN and UN remain highly vulnerable to all the attacks with a significant 37-40% drop in their average Dice score. In consensus to the classification results discussed earlier, comparing the three segmentation models only in terms of generalizability would not have been sufficient to determine the best one. Both its resilience to samples distorted with Rician noise and its consistent resistance to adversarial attacks make DN the strongest model among its competitors for this task.
In this paper, for the first time, we explored adversarial examples in medical imaging for the tasks of classification and segmentation. We propose a strategy for model evaluation by leveraging task-specific adversarial attacks, that evaluate not only a model’s generalizability, but also its robustness. We showed that for two models with comparable performance, their relative exploration of the underlying data manifold may have significant differences, hence resulting in varying robustness and model sensitivities. Specifically, we demonstrate that for segmentation tasks the use of dense blocks and skip connections contributes to both improved generalizability and robustness, while model depth seems to increase the resistance of classification models to adversarial examples.
-  C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, R. Fergus: Intriguing properties of neural networks. ICLR 2014
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, Z. Wojna: Rethinking the Inception Architecture for Computer Vision. CVPR 2016
-  O. Ronneberger, P. Fischer, T. Brox: U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI 2015
-  I. Goodfellow, J. Shlens, C. Szegedy: Explaining and Harnessing Adversarial Examples. ICLR 2015
-  S.M. Moosavi-Dezfooli, A. Fawzi, P. Frossard: DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks. CVPR 2016
-  N. Papernot, P. D. McDaniel, S. Jha, M. Fredrikson, Z. Berkay Celik, A. Swami: The Limitations of Deep Learning in Adversarial Settings. EuroS&P 2016
-  C. Xie, J. Wang, Z. Zhang, Y. Zhou, L. Xie, A. L. Yuille: Adversarial Examples for Semantic Segmentation and Object Detection. ICCV 2017
-  N. Papernot, P. D. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, A. Swami: Practical Black-Box Attacks against Machine Learning. AsiaCCS 2017
-  A.G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, H. Adam: MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. CoRR abs/1704.04861 2017
-  L. Ballerini, R.B. Fisher, R.B. Aldridge, J. Rees: A Color and Texture Based Hierarchical K-NN Approach to the Classification of Non-melanoma Skin Lesions. Color Medical Image Analysis 2013
-  V. Badrinarayanan, A. Kendall, R. Cipolla: SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(12): 2481–2495 2017
-  S. Jégou, M. Drozdzal, D. Vázquez, A. Romero, Y. Bengio: The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation. CVPR Workshops 2017
-  A. G. Roy, S. Conjeti, D. Sheet, A. Katouzian, N. Navab, C. Wachinger: Error Corrective Boosting for Learning Fully Convolutional Networks with Limited Data. MICCAI 2017
-  D.S. Marcus, T.H. Wang, J. Parker, J.G. Csernansky, J.C. Morris, R.L. Buckner: Open Access Series of Imaging Studies (OASIS): Cross-sectional MRI Data in Young, Middle Aged, Nondemented, and Demented Older Adults. J. Cognitive Neuroscience 19(9): 1498–1507 2007
-  B. Landman, S. Warfield: MICCAI Workshop on Multiatlas Labeling. In MICCAI Grand Challenge, 2012.
-  M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro et al.: TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. CoRR abs/1603.04467 2016
-  J. Rauber, W. Brendel, M. Bethge: Foolbox v0.8.0: A Python toolbox to benchmark the robustness of machine learning models. CoRR abs/1707.04131 2017