Melanoma is a highly aggressive form of skin tumor that can be successfully treated when diagnosed at an early stage . Suspicious symptoms can be detected visually, and for that reason computer-vision techniques applied to skin image analysis are becoming increasingly important as a feasible and inexpensive method for skin care .
For an effective interpretation of skin lesions, color is known to be a fundamental visual feature, together with size, shape and texture . In computerized skin image analysis, a representation of the image in terms of visual features is usually built in order to find relevant cues for diagnosis. Deep learning techniques, which can automate the feature design step, are outperforming traditional, hand-crafted features-based methods in most computer-aided skin disease diagnosis tasks [4, 5]. However, training these techniques requires large databases of labeled skin images. These databases often contain samples obtained at different hospitals, with different acquisition systems and under varying illumination conditions, which poses a complex challenge for robust image analysis.
Usually, the input of skin image understanding techniques undergoes a color normalization step, in order to achieve color constancy . This has been recently shown to be helpful for subsequent automatic diagnosis tasks . This procedure is related to the need of handcrafted feature-based methods to receive normalized input images. This is especially important at test time, since feeding the model images that are significantly different from those that the algorithms saw during training time can lead to a poor generalization of the method, due to their limited expressiveness. However, computational color constancy is a hard problem, and the output can sometimes be unpredictable.
In this work, we propose a data augmentation technique adapted for skin lesion analysis with deep neural networks, leveraging the fact that their virtually unlimited expressive power enables them to be trained with extensive variability in the data manifold, in order for them to learn features invariant to those factors of variation. Specifically, instead of relying on existing illuminant color compensation techniques to achieve a normalized dataset, we apply them to obtain an estimate of the color of the illuminant of the scene. These illuminants are then sampled at training time, and applied to white-balanced training images in order to generate new training images that simulate different but plausible illumination conditions at acquisition time. This augmentation technique leads to a robust training stage, which allows for promising results to be obtained both in segmentation and classification tasks. Our work is framed into the ISIC 2017 Challenge: Skin Lesion Analysis-Towards Melanoma Detection . The competition provides three different goals, from which we only consider skin lesion segmentation and classification.
Ii Data-Driven Artificial Data Augmentation
In order to train complex models with a small amount of data, image augmentation schemes are often applied. These consist of transforming input images into new but plausible examples that can help the network to generalize better. Although recent generative models suggest that realistic medical image generation is feasible by means of more complex techniques 
, typically basic linear transformations on the image geometry are performed, such as shifting, scaling or rotating. However, applying domain-specific knowledge on the expected images,e.g. on the image acquisition process, can help to generate a more diverse set of artificial images while keeping plausibility.
Below we describe a method to jointly normalize color in skin images and learn from the available data the distribution of scene illuminants. These illuminants are then applied to white-balanced images for generating useful artificial images.
Ii-a Color Image Augmentation via Illuminant Estimation
A well-known property of the human visual system is the ability to perceive the color of an object as roughly constant, even when the color of the light projected on the scene is modified. This effect is known as color constancy and, although not yet fully understood, it can be partially explained by the joint interaction of low level mechanisms that act on the response of the cones in the retina and by higher cognitive-level effects . In parallel, there exists a large number of techniques that attempt to reproduce such effect on image acquisition and processing systems within the field of computational color constancy. In this context, the goal is often to first estimate the illuminant of the scene, and then remove it from the input image by means of a chromatic adaptation transform. Usually this is achieved in commercial cameras by means of proprietary white-balancing algorithms. However, this general approach does not always work in a predictable manner, as the illuminant depends not only on the light cast on the scene but also on the image content.
A white-balancing technique mainly works by manipulating image intensities in such a way that the scene appears as being lit by a neutral illuminant. This is usually accomplished separately in the different spectral components of the image. Denoting an image as , a typical white balancing technique solves the following equation:
where and is the white-balanced version of , which can be recovered by inversion of Eq. (1). In this work, we estimate the illuminant on images from the training set applying the Shades of Gray color constancy technique , as implemented by the general color constancy framework described in . A sample of training images and their white-balanced versions is shown in Fig. 1, while Fig. 2 shows the full empirical distribution over the illuminants.
Note that in 
the same approach was successfully employed to normalize skin images before supplying them to a classifier, observing an improvement in performance after this normalization. In this work, we proceed in a substantially different direction. Instead of only removing illuminant variations, we collect the set of illuminants extracted from the training set,. At training time, we augment the white-balanced dataset by color-casting each corrected image with an illuminant extracted from , i.e.,
. This produces realistically rendered versions of each of them under a different illuminant. This technique is applied in an online fashion (randomly yielding one version of each image per epoch) and by applying a Von Kries-like diagonal chromatic adaptation transform. Notice that, if the illuminant is the same that was extracted from , this transformation reduces to the identity, i.e., . However, if comes from a different image, this transformation leads to a color-casted version of . The result of applying this procedure to a white-balanced image from the training set can be observed in Fig. 3. With this strategy, the method is able to realistically augment the original dataset, preventing overfitting when training a deep neural network and achieving a more robust training procedure.
After extracting the set of illuminants from the training data, we need to access its underlying probabilistic distribution in order to generate new plausible images. The way in which that distribution is modeled is relevant to the quality of the obtained artificial images. In , a similar technique was applied to augment the dataset before learning a deep model that can achieve color constancy on natural images. In that work, the authors applied a -means clustering to the retrieved illuminants, with
being an heuristically determined free parameter. However, this approach may lead to an over-representation of areas in the illuminant space that, although representing a separate cluster, do not contain a significant amount of samples. Here we apply a different strategy and directly sample from the raw empirical distribution of illuminants. This way, each time a training example is selected we randomly choose an illuminant from
with a uniform probability distribution, producing a new color-casted image.
Ii-B Gamma-Correction Augmentation
In addition to the white-balancing, a digital camera applies several other normalization steps . One of them is the well-known gamma correction. When a camera captures light input, theoretically the received signal is a linear function of photons reaching the device’s sensors. However, for a more natural luminance reproduction in digital screens, a non-linear transformation with a power function is usually applied:
where is the luminance reaching the camera, is the corresponding post-processed image ready for display, and is the correction constant. Unfortunately, the specifications of this transform depend heavily on the characteristics of the display and on the imaging device manufacturer. To compensate for this issue, we also applied gamma corrections to the training images both at train and test time, aiming for a more robust representation learning of our model. Thus, after undergoing the color transformation explained above, we randomly draw for each image and epoch a correction factor
from a Normal distribution of meantruncated at 0 and 2, and apply a power-law mapping similar to the one in Eq. (2).
Ii-C Non-Linear Geometrical Image Augmentation
To complement the color transformation detailed above, we applied standard geometrical (linear) data augmentation techniques, namely rotation, horizontal and vertical flipping, translation and scaling of the input image. Moreover, to account for the non-linear distortions typical of soft tissues such as skin, we applied several non-linear transformations of the image geometry, similar to those proposed in .
Iii Deep Neural Networks for Skin Lesion Segmentation
The described lesion segmentation problem is approached using a Convolutional Neural Network (CNN), coupled with extensive data augmentation. Data was augmented applying the transforms described in Section II and re-scaled before being employed as input to a U-Net. The U-Net architecture was first proposed in 
, and is a Fully Convolutional Network, originally designed for segmenting neuronal structures and cells in microscopy images. It is a powerful deep classifier, that can be successfully trained with a relatively low amount of training data and still produces accurate segmentations.
The architecture of the U-Net is represented in Fig. 4
. The network has two main paths: a contracting path, and a dimensionally symmetric expanding path. The contracting path consists of consecutive convolutional layers with a stride of
. Each convolutional layer is followed by a Rectified Linear Unit (ReLU) activation and batch normalization, except for the last layer, which is activated by a sigmoid activation function. The stride of the convolutional layers is selected so that the dimension of the output feature map of the contracting path layers decreases until, where is the number of filters. This point in the network marks the beginning of the expanding path. The output of each layer is upsampled so that it has the same dimension as the corresponding layer in the contracting path. To compensate for the loss in spacial resolution that results from the multiple downsampling operations, the upsampled feature map is concatenated with the feature map of the corresponding layer in the contracting path.
Iv Deep Neural Networks for Skin Lesion Classification
The image classification task in the ISIC 2017 Challenge comprises two independently evaluated binary classification subtasks over the same set of images: 1) discriminating melanoma vs. all the other lesions and 2) discriminating seborrheic keratosis from every other kind of deseases. Both tasks were independently approached by means of separately trained networks. Nevertheless, as a common strategy, and following the rationale in , we chose to leverage the outcome from the lesion segmentation task by feeding the classification stage with images cropped around the bounding box defined by the segmentation mask. At train time we used the ground truth masks and increased the tight bounding box by for incorporating information on the appearance of the skin around the lesion, which may contain useful information for diagnosis. At test time we relied on the predicted segmentations and applied the same margin, which can also help alleviate small segmentation errors and keep lesion border information.
We trained both tasks with layer versions of deep residual networks 
. These were initialized with the weights learned by pre-training over the ILSVRC2012 Imagenet database. We replaced the last fully connected layer of the pretrained network by a dropout stage and two dense layers ofand outputs, respectively, followed by the final softmax. Given the strong tendency to overfit, a dropout probability of was set, and a two-phase fine-tuning was performed. During the first stage, only the new dense layers were trained (using an Adadelta optimizer). During the second phase, the first
layers were still kept frozen, and the rest were trained using stochastic gradient descent with a learning rate ofand momentum (). The loss function was categorical cross entropy, but the losses associated with samples from each class were weighted differently, to compensate for class imbalance.
V Experimental Evaluation
Both models were tested on the ISIC 2017 Challenge. The organizers provided a training set of images with ground-truth for segmentation and classification, a validation set of images, and a test set with images. It is important to stress that we did not employ any external skin image set or categorical data (other than natural images from the Imagenet dataset in the Resnet50 pretraining). This was allowed by the challenge organizers, but we wanted to explore the capability of the proposed technique to improve results when only few domain-specific images are available for training.
For performing predictions on unseen images, we applied randomly selected illuminants to the white-balanced input image, and predicted on several color casted versions of it. The overall prediction for each pixel was computed as the median of the predictions on each tested image. No further post-processing was performed in the output.
Table I shows the results obtained for the segmentation task on the validation set, while Table II does so for the classification task. Results on the test set will be supplied for both tasks as soon as they are available.
Acc = Accuracy, DC = Dice Coefficient, JD = Jaccard Distance
Ss = Sensitivity, Sp = Specificity
|3.2) Seborrheic keratosis||0.954||0.867||0.898||0.857||0.870|
AUC = Area Under the ROC curve, AP = Average Precision
ACC, AP, SS and SP were computed at 0.5 confidence threshold
V-a Discussion and Conclusion
In this paper we presented our approach to the Segmentation and classification tasks of the ISIC 2017 Challenge Skin Lesion Analysis - towards melanoma detection. We focused our efforts on exploiting the information obtained from applying color constancy algorithms to the training set in order to perform extensive data augmentation that could help regularize our convolutional neural networks-based learning pipeline. The obtained segmentations were leveraged to improve the full image classification, obtaining competitive results, especially on the segmentation task.
The strong shift of the distribution in Fig. 2 towards the quadrant suggests that part of the illuminant estimation may be inaccurately considering the skin’s color as being produced by an unnatural reddish illuminant in some cases. In the future, we aim at further investigating this effect.
-  D. Meckbach, J. Bauer, A. Pflugfelder, F. Meier, C. Busch, T. K. Eigentler, D. Capper, A. von Deimling, M. Mittelbronn, S. Perner, K. Ikenberg, M. Hantschke, P. Büttner, C. Garbe, and B. Weide, “Survival According to BRAF-V600 Tumor Mutations – An Analysis of 437 Patients with Primary Melanoma,” PLOS ONE, vol. 9, no. 1, pp. 1–10, 2014.
-  M. Celebi, T. Mendonca, and J. Marques, Eds., Dermoscopy Image Analysis, ser. Digital Imaging and Computer Vision. CRC Press, Sep. 2015, dOI: 10.1201/b19107.
-  N. R. Abbasi, H. M. Shaw, D. S. Rigel, R. J. Friedman, W. H. McCarthy, I. Osman, A. W. Kopf, and D. Polsky, “Early diagnosis of cutaneous melanoma: Revisiting the ABCD criteria,” vol. 292, no. 22, pp. 2771–2776.
-  N. Codella, Q.-B. Nguyen, S. Pankanti, D. Gutman, B. Helba, A. Halpern, and J. R. Smith, “Deep Learning Ensembles for Melanoma Recognition in Dermoscopy Images,” arXiv:1610.04662 [cs], Oct. 2016.
-  L. Yu, H. Chen, Q. Dou, J. Qin, and P. A. Heng, “Automated Melanoma Recognition in Dermoscopy Images via Very Deep Residual Networks,” IEEE Transactions on Medical Imaging, vol. PP, no. 99, pp. 1–1, 2016.
-  A. Gijsenij, T. Gevers, and J. v. d. Weijer, “Computational Color Constancy: Survey and Experiments,” IEEE Transactions on Image Processing, vol. 20, no. 9, pp. 2475–2489, Sep. 2011.
-  C. Barata, M. E. Celebi, and J. S. Marques, “Improving dermoscopy image classification using color constancy,” IEEE journal of biomedical and health informatics, vol. 19, no. 3, pp. 1146–1152, May 2015.
-  “International Skin Imaging Collaboration (ISIC) Project.” [Online]. Available: http://isdis.net/isic-project/.
-  P. Costa, A. Galdran, M. I. Meyer, M. D. Abràmoff, M. Niemeijer, A. M. Mendonça, and A. Campilho, “Towards Adversarial Retinal Image Synthesis,” arXiv:1701.08974 [cs, stat], Jan. 2017, arXiv: 1701.08974. [Online]. Available: http://arxiv.org/abs/1701.08974
-  M. D. Fairchild, Color Appearance Models, 3rd ed. Wiley.
-  G. D. Finlayson and E. Trezzi, “Shades of Gray and Colour Constancy,” Color and Imaging Conference, vol. 2004, no. 1, pp. 37–41, Jan. 2004.
-  J. van de Weijer, T. Gevers, and A. Gijsenij, “Edge-based color constancy,” IEEE transactions on image processing: a publication of the IEEE Signal Processing Society, vol. 16, no. 9, pp. 2207–2214, Sep. 2007.
-  J. von Kries, “Influence of adaptation on the effects produced by luminous stimuli,” pp. 109–119.
-  Z. Lou, T. Gevers, N. Hu, and M. P. Lucassen, “Color Constancy by Deep Learning.” British Machine Vision Association, 2015, pp. 76.1–76.12.
-  A. R. Robertson, “The CIE 1976 Color-Difference Formulae,” Color Research & Application, vol. 2, no. 1, pp. 7–11, Mar. 1977. [Online]. Available: http://onlinelibrary.wiley.com/doi/10.1002/j.1520-6378.1977.tb00104.x/abstract
-  M. Bertalmío, Image Processing for Cinema, 1st ed. Boca Raton: Chapman and Hall/CRC, Feb. 2014.
-  O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” arXiv:1505.04597 [cs], May 2015, arXiv: 1505.04597.
-  B. Kayalibay, G. Jensen, and P. van der Smagt, “CNN-based Segmentation of Medical Imaging Data,” arXiv:1701.03056 [cs], Jan. 2017, arXiv: 1701.03056.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2016, pp. 770–778.