Skin lesion analysis through image processing and machine learning techniques has attracted a lot of attention in the last decades. With the advent of deep convolutional neural networks (CNNs) and different scientific competitions on this topic in the recent years, numerous powerful computational methods have been emerged to solve different problems in this area. In general, the final goal of using Computer Aided Diagnosis (CAD) systems for skin lesion analysis is to automatically detect any abnormalities or skin diseases. The Methods in the literature usually follow a certain analysis pipeline 
: first, lesion area in the image is detected (segmentation task) helping other computerized algorithms to extract discrimination descriptors (feature extraction task) from the lesion in the second step, in order to finally decide about the type of skin lesions (classification task).
Although modern CNNs are able to extract relevant features from the raw dermoscopic images and classify them without the need of lesions or their attributes segmentation map, it has been shown that incorporating that information can significantly improve lesion classification task[3, 4]. Yu et al.  showed that incorporating lesion mask can elevate lesion classification and they achieved 1st rank in the ISIBI2016: Melanoma Recognition challenge. Moreover, Gonzalez Diaz 
proposed a lesion classification framework that incorporates both lesion segmentation and attributes probability maps. He was able to achieve state-of-the-art results using only one single model by adding this additional information. Such researches and their promising results prove the importance of lesion mask and its attributes segmentation tasks.
There are several methods introduced in the literature to address lesion segmentation. Some of them are based on thresholding algorithms , some followed region based methods  and researches that proposed a hybrid method by combining thresholding and active contours . Dealing with lesion segmentation as a saliency detection problem has also been exploited several times . However, the most recent successful approaches were based on deep convolutional neural networks (CNNs) [9, 3, 4, 10]. To have a more comprehensive review of different lesion segmentation algorithms, enthusiastic readers can refer to .
Lesions attributes are meaningful visual patterns in the lesion texture. Based on the lesion type and malignancy severity, attributes appearance can change. Most lesions may not show any pattern similar to a specific attribute, some may contain textural pattern related to several attributes. Attribute types that we are interested in are pigment networks , globules/dot , Milia like cysts, negative networks , and streaks . To the best of knowledge, unlike lesion segmentation, there is a little literature around segmentation of attributes in dermoscopic images. The main reason for this is lack of annotated data. However, after the first International Skin Imaging Collaboration (ISIC) challenge , some approaches have been proposed for this task. Kawahara et al.  proposed a segmentation model based on CNNs which incorporated a novel multi-channel Dice loss to predict 5 attributes all together in his proposed network. They were able to achieve first rank in ISIC2017-Task2 attributes detection. Unlike  which tried to train a network predicting all 5 attribute once together, in this paper, we propose to use separate segmentation networks for lesion and each attribute. Here, we are looking for powerful CNN models to segment lesion and its related attributes from dermoscopic images. In order to avoid the hurdles during training a CNN from scratch, we propose to use transfer learning in a novel segmentation framework. We also carefully tailored simple and yet effective loss functions for each task of lesion and attribute segmentation. Another novel aspect of the present work is extensive use of augmentation routines. Some of these augmentations are specially designed to imitate the appearance variation presented in dermoscopy images. Prediction maps are then generated by ensembling the outputs of different models and different test time augmentation. To obtain the binary mask of the lesion or dermoscopy attribute, some optimal thresholding and post-processing tasks are applied on the final ensemble prediction map.
comprises some explanation about the dataset, evaluation metrics, and implementation details. Experimental results on ISIC2018 dataset are reported in sectionIV, and finally, the conclusion is drawn in section V.
Ii-a Transfer learning
Training deep convolutional neural networks from scratch is a challenging task due to the effects of various factors on the final outcome . These factors are, designing a robust network architecture, appropriate weight initialization, setting an optimum training strategy (opting loss function, learning rate, weight decay, etc.), and a powerful hardware to support the computational complexity of CNNs. Furthermore, providing a large dataset is an important requirement for CNNs to take the most advantage of their deep structures. However, in many circumstances particularly in the medical domain, annotated data is often scarce and expensive to acquire .
To circumvent these hurdles, transfer learning has been widely utilized in the literature . Transfer learning is about using the knowledge learned from dealing with one problem to address another (often related) problem 
. A common usage of transfer learning is to fine-tune powerful pre-trained deep learning models for the desired task. In this paper, we propose to use deep pre-trained models in a multi-scale framework for segmentation task.
Our proposed segmentation models are based on the well-known UNet  structure. The original UNet  architecture comprises an encoding path to extract abstract feature maps from the input images and a decoding path to translate the extracted feature maps into output segmentation map. In order to keep the high-resolution information in the decoding path and have a more stable training process, UNet  utilizes skip connections between corresponding levels of encoding and decoding path. Here, we use a variant of the UNet architecture to boost the performance on segmentation tasks.
Firstly, for feature extraction in the encoding path of our proposed segmentation models, pre-trained models (on ImageNet dataset) are incorporated. The pre-trained networks that we utilized in this framework are 152-layered ResNet (ResNet152) , 169-layered DensNet (DensNet169) , Xception , and Inception-ResNet v2 (ResNetV2) 
. These models are referred to as the base model in our proposed segmentation framework. Outputs of the last convolutional layer from the base network are extracted and up-sampled followed by convolution layers in each level of decoding path. The number of up-sampling layers in this path is equal to the number of max-pooling layers in the encoding path. Skip connections are drawn between the same levels of encoding and decoding path, however, unlike the original UNet, they are merged by ”add” operators. In order to equalize the number of feature maps in the corresponding levels, bottleneck convolutions have been employed. Feature maps from different decoding levels are then gathered in a pyramid feature pooling manner. Afterward, a series of convolutions are applied to them. The final output of the network (segmentation map) is generated by applying a 11 convolutions followed by a sigmoid activation on the output of the last convolutional layer. The schematic architecture of the proposed transfer learning model is shown in Fig. 1.
Apart from UNet like networks, we incorporated DeepLabV3  network for our segmentation tasks as well. DeeplabV3 is an atrous convolution enhanced network, originally designed for semantic segmentation. The only modification that we have made to the DeepLabV3 network is that we changed its last convolutional layer to get only one segmentation map in the output. Like other networks, the output layer consists of a 1x1 convolution with sigmoid activation. It is worth mentioning that our pre-trained weights for DeepLabV3, was originally obtained based on Pascal dataset .
Images are preprocessed according to the type of base network that has been used. Different preprocessing schemes were applied when they were trained on the ImageNet dataset. Therefore, we also incorporate the same preprocessing procedures that were originally deployed for each individual base model.
If the base network is one of the ResNet152 or ResNetV2 the preprocessing procedure comprises subtracting the channel wise mean of the ImageNet dataset from each image. If the base model is DensNet169, mean image value of the ImageNet dataset is subtracted from images and the results are divided by the standard deviation of it. For the Xception based model, image values are scaled between (-1,1) and for the DeeplLabV3, input images are scaled between (0, 1). Additionally, for lesion and attribute segmentation tasks, images are resized to 192256 and 384576, respectively.
Ii-A3 Data augmentation
To eliminate the effect of the small dataset, an extensive set of augmentations has been incorporated. All augmentations were carefully designed to simulate all the possible variations and transformation present in dermoscopic images. Explaining the details of all incorporated augmentation routines is beyond the scope of this manuscript. However, a curated list of employed augmentations are: Random vertical and horizontal flips, random rotation in the range of 0 to 40 degrees, image zooming with a random scale between 0.7 to 1.3, random image translation, random image shearing to extent of 0.3, random shift in color channels (change image color), scaling image intensities with a random scale in the range of 0.7 to 1.3, and adding noises from different random distribution and types (Gaussian, Speckle, Salt & Pepper).
Furthermore, we have introduced four new routines to augment dermoscopic images, which can imitate dermoscopic images variations more realistically. These new augmentations are as follow: randomly contrast adjustment (decreasing and increasing image contrast using linear histogram equalization), random sharpness adjustment (blurring using the Gaussian filter of different sizes and sharpening by employing unsharp masking technique). Randomly disturbing the image illumination uniformity (images are multiplied by an illumination gradient map, the gradient map can be either axial or radial). Randomly hair occlusion (using a hair simulator toolbox , masks of hairs with different properties and appearance are made). Note that all of the above-mentioned augmentations are applied on the fly during training.
Ii-B Lesion Segmentation
Ii-B1 Loss function
We propose a novel yet simple Loss function to target lesion segmentation task. Our Loss function is a combination of soft Jaccard index with binary cross entropy (BCE).
In which, is network prediction, is ground truth segmentation, and
is the number of pixels. The Jaccard part of loss function makes the training process converge faster and more stable. It also, robust against the pixels’ imbalanced class population. Jaccard loss is appropriate to learn general parts of the lesion in the segmentation output whereas BCE penalizes the loss for pixel-wise errors. Therefore, it leads to a more precise boundary estimation and more soft predictions on uncertain areas. For lesion segmentation, we set the coefficientand .
Ii-B2 Training procedure
All decoding layers of the proposed segmentation network are initialized using “uniform Glorot”  initialization method. Loss optimization has been carried out using AMSGrad variant of Adam optimizer  with the learning rate and weighs decay parameters equal to
for all layers. A 5-fold cross-validation scheme has been utilized with 70 epochs for each fold. The best performing model on the validation subset is saved during the training process based on the loss value.
Ii-B3 Post processing
For acquiring the final binary mask, first, the lesion marker () is constructed by applying threshold () on the prediction map. Lesion mask () is also obtained through thresholding the prediction map with upper value than (). Afterward, () is refined to consist only an object with the biggest area. Consequently, the segmentation mask of the lesion is constructed through binary morphological reconstruction :
The threshold values for and are chosen based on a grid search on different values in the ranges and .
Ii-C Lesion attributes segmentation
Attributes segmentation is a more challenging task than lesion segmentation due to their small structures and low variance in their appearance. Furthermore, the population of different attributes in the data set is very uneven. For instance, pigment network is an attribute that is present in the most of dermoscopic images whereas streaks are very rare attributes. A few attempts in the literature have been made to address the problem of lesions’ attribute segmentation. In this work, unlike, we proposed to use separate models to segment each attribute. In other words, for detecting each attribute, we propose to train a specific model. Models for these tasks are the same as the models we introduced in the previous section for segmenting the lesion mask, however, we are predicting maps of different attributes rather than predicting lesion masks.
Ii-C1 Data Subsampling
As we mentioned in the previous section, we address all attributes segmentation as a separate binary segmentation task. To deal with the unbalanced class in that binary segmentation task (which is very crucial when we are looking for attributes like Streaks that has a considerably smaller share of the dataset), we propose to use subsampling of negative class.
For clarification, an example of subsampling procedure for streak attribute is given in the following. Out of 2594 images in the ISIC2018 dataset , 100 images contain Streaks attribute. To deal with imbalanced class in the data, we randomly sample 100 images from negative class (remaining 2494 image without Streaks). Although the number of data is equally divided between the two classes, many training data are ignored for this purpose. Nevertheless, the class adjustment leads to having a sensible prediction map.
Ii-C2 Loss Function
A modified Jaccard loss function  is used for attribute segmentation:
This loss function is able to predict a smoother segmentation output. However, to prevent network to be biased toward negative class and to clearly predict all zero pixels, we proposed to set smoothing coefficients and as recommended in .
Ii-C3 Training procedure
The training procedure for segmenting each attribute is done just like the process we explained for lesion segmentation (II-B2).
Post-processing procedures for attributes consist of restricting the prediction map to the lesion area (by multiplying the attribute predicted map into lesion segmentation map), finding the high and low thresholding values () to construct plausible and , and generating final attribute segmentation using morphological reconstruction. Note that unlike lesion segmentation task, here we do not remove any objects from constructed marker by area analysis.
Ii-D Test time augmentation and prediction ensembling
It has been shown that ensembling techniques can elevate the performance considerably , therefore we tried to use prediction ensembling as much as possible. Various test time augmentations are applied and then their outputs are merged (averaging). These ensembled predictions are then going under post-processing routines.
For all segmentation models, test images are predicted using all trained weights in 5 folds experiments. In each fold, the original image is augmented with four different transformations: horizontal flip, vertical flip, contrast enhancing of a rotated version of the image, and sharpening the image. Hence, to construct the final prediction from a single model in our proposed approach, 25 predictions are acquired and averaged. Moreover, the final prediction maps from different models are merged together to obtain the ultimate prediction.
Iii Experimental setups
Iii-a Data sets
The data set used for training and validating of the proposed algorithms was adopted from the “ISIC 2018: Skin Lesion Analysis Towards Melanoma Detection” grand challenge datasets [1, 28]. The current dataset is related to task 1 and 2 of the ISIC2018 challenge and comprises 2594 images with lesion segmentation and ground truth of five dermoscopic attributes.
There is another validation set with 100 images and a test set comprising of 1000 images. The ground truths for the validation and test sets are not released by the time this paper is written. These two latter data sets are used to evaluate and rate different approaches participated in the challenge.
Iii-B Evaluation metrics
Evaluation metrics for lesion segmentation tasks are thresholded Jaccard index, row Jaccard index, sensitivity, specificity, accuracy, and Dice coefficient . However, the selected metric for ranking participant in task1 (lesion segmentation) is thresholded Jaccard index (averaged over all test images).
In the second task (attributes segmentation), each attribute is evaluated separately in the way that all output segmentation maps from all test image are concatenated together to form a multi-channel output with a specific size (number of test images), and it is then evaluated using the Jaccard index and Dice coefficient. The final evaluation of all attributes is drawn out by averaging the attained metrics for all 5 different attributes.
|Base Network||()||Thresholded Jaccard||Jaccard||Dice||Accuracy||Sensitivity||Specificity|
|DeepLab v3||(0.8, 0.5)||0.7932||0.8338||0.9030||0.9655||0.9115||0.9732|
|Base Network||()||Thresholded Jaccard|
|DeepLab v3||(0.8, 0.65)||0.825|
Iii-C Implementation details
Model training, validation and prediction have been implemented using Keras library
with Tensorflow backend. Grid search on thresholds, prediction ensembling, post-processing have been done with the help of Mathwork Matlab 2016b. All experiments were done on an Intel Core i9 machine equipped with 128 GB of RAM and two GPUs (Nvidia Geforce GTX 1080 Ti) running on windows. Moreover, we have adopted our multi-scale pyramid feature pooling segmentation framework fromhttps://github.com/selimsef/dsb2018_topcoders/, and DeepLabV3 from https://github.com/bonlime/keras-deeplab-v3-plus.
Iv Results and Discussion
Iv-a Lesion segmentation results
Table I shows the lesion segmentation evaluation results extracted from 5 fold cross-validation experiments on 2549 training images using different base models. It is inferred from Table 1 that ”resnetv2” outperforms other models. To elevate segmentation results, we ensemble predictions from different models before post-processing. Evaluation metrics derived from ensemble predictions are better than every single model performance. Using the obtained optimal threshold on cross-validation and threshold grid search experiments, we construct the final segmentation outputs for ISIC2018-Task1 validation and test sets. The results averaged on 100 images of validation set is reported in Table II.
|Base Network||Pigment Networks||Globules||Milia like Cysts||Negative Networks||Streaks||Average|
To show the power of the proposed segmentation method and the positive effect of model ensembling, outputs of segmentation models and the final segmentation mask are illustrated for five samples in the Fig. 2.
In the first row of Fig. 2 original images are depicted with the boundaries of lesion overlaid on them. In that images, green contours represent ground truth boundaries, and red contours show the boundaries of proposed segmentation framework. The second to seventh rows of Fig. 2 illustrate the output predictions of segmentation framework with DensNet169, ResNet152, ResNetV2, Xception, and DeepLab, respectively. The final row in the Fig. 2 is the ensemble prediction obtained by averaging all the above predictions.
As illustrated in Fig. 2, DenseNet169 shows a good performance on three samples (column 3rd to 6th) but it does not perform well on the first two columns (left to right), whereas other networks performed well enough on those two samples. Therefore, by benefiting from the ensembling, we could achieve a plausible result in the last row of Fig. 2. The same phenomenon happened in all samples, where ensembling outperformed each individual model.
Iv-B Lesion’s attributes segmentation results
For attributes segmentation, we have evaluated each trained model on ISIC2018-Task2 training set (cross-validation experiment). Obtained results are summarized in the Tables III. As inferred from the table, ensemble outputs from different models outperformed every single model for different attribute segmentation too.
Output segmentations of different attributes for five sample dermoscopic images are illustrated in Fig. 3, in which, each column shows a sample and each row is associated with one specific attribute. Order of attributes from the first row to the last row in Fig. 3 is as follows: globules, Milia-like cysts, negative network, pigment network, and streaks. In each image of Fig. 3, ground truth of attributes are overlaid with green color on the image, proposed segmentations are depicted with red color, and their intersection is specified by orange color. As one can see, the proposed attribute segmentation algorithm is able to successfully predict regions with different attribute labels (intersection regions are rather large).
There are also three submissions for the ISIC2018-Task 2. Submissions are based on the ensemble of predictions from three base models (ResNet152, DenseNet169, and ResNetV2). In every submission, optimal thresholds for each attribute were utilized (optimal thresholds are obtained based on cross-validation experiments on each attribute) to elevate the performance. The first submission used all networks for all attributes and applied the lesion mask restriction on its prediction in the post-processing tasks. The second submission was like the first one, but without applying lesion mask restriction. The third submission was similar to the first submission (with skin lesion restriction applied), but with this difference that thresholds for all attributes in that submission are decreased in amount of 0.05 in order to keep more objects in the .
In this paper, we proposed to use transfer learning and extensive ensemble of predictions in order to boost performance on the skin lesion boundaries and its attributes segmentation task. Proposed segmentation framework incorporates a variant of UNet  architecture with pre-trained base networks in its encoding path, and a pyramid feature pooling in its decoding path. We also utilized several on-line data augmentation techniques, some of which were specially tailored to mimic the variation present in the dermoscopic images.Moreover, Two modified loss functions were introduced according to each segmentation task requirements. Our models have been evaluated in a 5-fold cross-validation framework to reduce the risk of over-fitting and generating a basis for optimal hyper-parameters estimation. Evaluation experiments on the training and validation set shows promising results for both lesion boundaries segmentation and lesion attribute detection tasks.
-  N. C. Codella, D. Gutman, M. Emre Celebi, B. Helba, M. A. Marchetti, S. W. Dusza, A. Kalloo, K. Liopyris, N. Mishra, H. Kittler, and A. Halpern, “Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic),” arXiv preprint arXiv:1710.05006, 2017.
-  N. Zamani Tajeddin and B. Mohammadzadeh Asl, “Melanoma recognition in dermoscopy images using lesion’s peripheral region information,” Computer Methods and Programs in Biomedicine, vol. 163, pp. 143 – 153, 2018.
-  L. Yu, H. Chen, Q. Dou, J. Qin, and P.-A. Heng, “Automated melanoma recognition in dermoscopy images via very deep residual networks,” IEEE transactions on medical imaging, vol. 36, no. 4, pp. 994–1004, 2017.
-  I. G. Díaz, “Dermaknet: Incorporating the knowledge of dermatologists to convolutional neural networks for skin lesion diagnosis,” accepted to be appeared in IEEE Journal of Biomedical and Health Informatics, 2018.
-  M. Emre Celebi, Q. Wen, S. Hwang, H. Iyatomi, and G. Schaefer, “Lesion border detection in dermoscopy images using ensembles of thresholding methods,” Skin Research and Technology, vol. 19, no. 1, pp. e252–e258, 2013.
-  M. Emre Celebi, H. A. Kingravi, H. Iyatomi, Y. Alp Aslandogan, W. V. Stoecker, R. H. Moss, J. M. Malters, J. M. Grichnik, A. A. Marghoob, H. S. Rabinovitz, et al., “Border detection in dermoscopy images using statistical region merging,” Skin Research and Technology, vol. 14, no. 3, pp. 347–353, 2008.
-  N. Zamani Tajeddin and B. Mohammadzadeh Asl, “A general algorithm for automatic lesion segmentation in dermoscopy images,” in Biomedical Engineering and 2016 1st International Iranian Conference on Biomedical Engineering (ICBME), 2016 23rd Iranian Conference on, pp. 134–139, IEEE, 2016.
-  M. Jahanifar, N. Zamani Tajeddin, B. Mohammadzadeh Asl, and A. Gooya, “Supervised saliency map driven segmentation of lesions in dermoscopic images,” accepted to be appeared in IEEE Journal of Biomedical and Health Informatics, 2018.
-  Y. Yuan and Y.-C. Lo, “Improving dermoscopic image segmentation with enhanced convolutional-deconvolutional networks,” arXiv preprint arXiv:1709.09780, 2017.
-  A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun, “Dermatologist-level classification of skin cancer with deep neural networks,” Nature, vol. 542, no. 7639, p. 115, 2017.
-  R. B. Oliveira, E. Mercedes Filho, Z. Ma, J. P. Papa, A. S. Pereira, and J. M. R. Tavares, “Computational methods for the image segmentation of pigmented skin lesions: a review,” Computer methods and programs in biomedicine, vol. 131, pp. 127–141, 2016.
-  J. Kawahara and G. Hamarneh, “Fully convolutional neural networks to detect clinical dermoscopic features,” accepted to be appeared in IEEE Journal of Biomedical and Health Informatics, 2018.
-  G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. van der Laak, B. Van Ginneken, and C. I. Sánchez, “A survey on deep learning in medical image analysis,” Medical image analysis, vol. 42, pp. 60–88, 2017.
-  A. Van Opbroek, M. A. Ikram, M. W. Vernooij, and M. De Bruijne, “Transfer learning improves supervised image segmentation across imaging protocols,” IEEE transactions on medical imaging, vol. 34, no. 5, pp. 1018–1030, 2015.
-  K. Weiss, T. M. Khoshgoftaar, and D. Wang, “A survey of transfer learning,” Journal of Big Data, vol. 3, no. 1, p. 9, 2016.
-  O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention, pp. 234–241, Springer, 2015.
-  J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 248–255, Ieee, 2009.
-  K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
-  G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks.,” in CVPR, vol. 1, p. 3, 2017.
-  F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” arXiv preprint, pp. 1610–02357, 2017.
C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning.,” inAAAI, vol. 4, p. 12, 2017.
-  L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” arXiv preprint arXiv:1802.02611, 2018.
-  M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” International Journal of Computer Vision, vol. 88, pp. 303–338, June 2010.
-  H. Mirzaalian, T. K. Lee, and G. Hamarneh, “Hair enhancement in dermoscopic images using dual-channel quaternion tubularness filters and mrf-based multilabel optimization,” IEEE Transactions on Image Processing, vol. 23, no. 12, pp. 5486–5496, 2014.
X. Glorot and Y. Bengio, “Understanding the difficulty of training deep
feedforward neural networks,” in
Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249–256, 2010.
-  S. J. Reddi, S. Kale, and S. Kumar, “On the convergence of adam and beyond,” 2018.
-  R. C. Gonzalez, R. E. Woods, and S. L. Eddins, “Morphological reconstruction,” Digital Image Processing using MATLAB, MathWorks, 2010.
-  P. Tschandl, C. Rosendahl, and H. Kittler, “The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,” Sci. Data, vol. 5, p. 180161, 2018.
-  N. C. Codella, Q.-B. Nguyen, S. Pankanti, D. Gutman, B. Helba, A. Halpern, and J. R. Smith, “Deep learning ensembles for melanoma recognition in dermoscopy images,” IBM Journal of Research and Development, vol. 61, no. 4, pp. 5–1, 2017.
-  F. Chollet et al., “Keras.” https://keras.io, 2015.