Artificial Intelligence for Diagnosis of Skin Cancer: Challenges and Opportunities

11/26/2019 ∙ by Manu Goyal, et al. ∙ Dartmouth College 79

Recently, there has been great interest in developing Artificial Intelligence (AI) enabled computer-aided diagnostics solutions for the diagnosis of skin cancer. With the increasing incidence of skin cancers, low awareness among a growing population, and a lack of adequate clinical expertise and services, there is an immediate need for AI systems to assist clinicians in this domain. A large number of skin lesion datasets are available publicly, and researchers have developed AI solutions, particularly deep learning algorithms, to distinguish malignant skin lesions from benign lesions in different image modalities such as dermoscopic, clinical, and histopathology images. Despite the various claims of AI systems achieving higher accuracy than dermatologists in the classification of different skin lesions, these AI systems are still in the very early stages of clinical application in terms of being ready to aid clinicians in the diagnosis of skin cancers. In this review, we discuss advancements in the digital image-based AI solutions for the diagnosis of skin cancer, along with some challenges and future opportunities to improve these AI systems to support dermatologists and enhance their ability to diagnose skin cancer.



page 6

page 8

page 13

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

According to the Skin Cancer Foundation, the global incidence of skin cancer continues to increase Foundation (2017)

. In 2019, it is estimated that 192,310 cases of melanoma will be diagnosed in the United States

Street (2019). However, the most common forms of skin cancer are non-melanocytic, such as Basal Cell Carcinoma (BCC) and Squamous Cell Carcinoma (SCC). Non-melanoma skin cancer is the most commonly occurring cancer in men and women, with over 4.3 million cases of BCC and 1 million cases of SCC diagnosed each year in the United States, although these numbers are likely to be an underestimate Rogers et al. (2015). Early diagnosis of skin cancer is a cornerstone to improving outcomes and is correlated with 99% overall survival (OS). However, once disease progresses beyond the skin, survival is poor Bray et al. (2018); Apalla et al. (2017).

In current medical practice, dermatologists examine patients by visual inspection with the assistance of polarized light magnification via dermoscopy. Medical diagnosis often depends on the patient’s history, ethnicity, social habits and exposure to the sun. Lesions of concern are biopsied in an office setting, submitted to the laboratory, processed as permanent paraffin sections, and examined as representative glass slides by a pathologist to render a diagnosis.

AI-enabled computer-aided diagnostics (CAD) solutions are poised to revolutionize medicine and health care, especially in medical imaging. Medical imaging, including ultrasound, computed tomography (CT), and magnetic resonance imaging (MRI), is used extensively in clinical practice. In the dermatological realm, dermoscopy or, less frequently, confocal microscopy, allows for more detailed in vivo visualization of lesioned features and risk stratification Hosny et al. (2018); Wang et al. (2016); Yap et al. (2018b); Goyal ; Würfl et al. (2016). In various studies, AI algorithms match or exceed clinician performance for disease detection in medical imaging Rajpurkar et al. (2017); Esteva et al. (2017). Recently, deep learning has provided various end-to-end solutions in the detection of abnormalities such as breast cancer, brain tumors, lung cancer, esophageal cancer, skin lesions, and foot ulcers across multiple image modalities of medical imaging Yap et al. (2018a); Havaei et al. (2017); Wei et al. (2019a); Tomita et al. (2019); Goyal et al. (2018).

Over the last decade, advances in technology have led to greater accessibility to advanced imaging techniques such as 3D whole body photoimaging/scanning, dermoscopy, high-resolution cameras, and whole-slide digital scanners that are used to collect high-quality skin cancer data from patients across the world Daanen and Ter Haar (2013); Ching et al. (2018). The International Skin Imaging Collaboration (ISIC) is a driving force that provides digital datasets of skin lesion images with expert annotations for automated CAD solutions for the diagnosis of melanoma and other skin cancers. A wide research interest in AI solutions for skin cancer diagnosis is facilitated by affordable and highspeed internet, computing power, and secure cloud storage to manage and share skin cancer datasets. These algorithms can be scalable to multiple devices, platforms, and operating systems, turning them into modern medical instruments Bengio et al. (2017).

The purpose of this review is to provide the reader with an update on the performance of artificial intelligence algorithms used for the diagnosis of skin cancer across various modalities of skin lesion datasets, especially in terms of the comparative studies on the performance of AI algorithms and dermatologists/ dermatopathologists. We dedicated separate sub-sections to arrange these studies according to the types of imaging modality used, including clinical photographs, dermoscopy images, and whole-slide pathology scanning. Specifically, we seek to discuss the technical challenges in this domain and opportunities to improve the current AI solutions so that they can be used as a support tool for clinicians to enhance their efficiency in diagnosing skin cancers.

2 Artificial Intelligence for Skin Cancer

The major advances in this field came from the work of Esteva et al. Esteva et al. (2017) who used a deep learning algorithm on a combined skin dataset of 129,450 clinical and dermoscopic images consisting of 2,032 different skin lesion diseases. They compared the performance of a deep learning method with 21 board-certified dermatologists for classification and differentiation of carcinomas versus benign seborrheic keratoses; and melanomas versus benign nevi. The performance of AI was demonstrated to be on par with dermatologists’ performance for skin cancer classification. Three main types of modalities are used for the skin lesion classification and diagnosis in the work described here: clinical images, dermoscopic images, and histopathology images. In this section, we start with analysis of publicly available skin lesion datasets, and then we provide different sub-sections dedicated to the artificial intelligence solution related to each type of imaging modality.

2.1 Publicly Available Datasets for Skin Cancer

  1. ISIC Archive: The ISIC archive gallery consists of many clinical and dermoscopic skin lesion datasets across the world such as ISIC Challenges datasets Codella et al. (2018), HAM10000 Tschandl et al. (2018), and BCN20000 Combalia et al. (2019).

  2. Interactive Atlas of Dermoscopy Argenziano et al. (2000): The Interactive Atlas of Dermoscopy has 1,000 clinical cases (270 melanomas, 49 seborrheic keratoses), each with at least two images: dermoscopic, and close-up clinical. It’s available for research purposes and has a fee of 250 Euros.

  3. Dermofit Image Library dermofit: The Dermofit Image Library consists of 1,300 high-resolution images with 10 classes of skin lesions. There is a need for a licensing agreement with a one-off license fee of 75 Euros, and an academic license is available.

  4. PH2 Dataset Mendonça et al. (2013): The PH2 Dataset has 200 dermoscopic images (40 melanoma and 160 nevi cases). It is freely available after signing a short online registration form.

  5. MED-NODE Dataset Giotis et al. (2015): It consists of 170 clinical images (70 melanoma and 100 nevi cases). This dataset is freely available to download for research.

  6. Asan Dataset Han et al. (2018a, b): It is a collection of 17,125 clinical images of 12 types of skin diseases found in asian people. It is freely available to download for research.

  7. Hellym Dataset Han et al. (2018a): This dataset consists of 125 clinical images of BCC cases.

  8. SD-198 Dataset Yang et al. (2018): The SD-198 dataset is the clinical skin lesion dataset containing 6, 584 clinical images of 198 skin diseases. This dataset was captured with digital cameras and mobile phones.

  9. SD-260 Dataset Yang et al. (2019): This dataset is a more balanced dataset when compared to the previous SD-198 dataset since it controls the class size distribution with preservation of 10–60 images for each category. It consists of 20,600 images with 260 skin diseases.

  10. Dermnet NZ 14: Dermnet NZ has one of the largest and most diverse collections of both clinical, dermoscopic and histology images of various skin diseases. These images can be used for academic research purposes. They have additional high-resolution images for purchase.

  11. Derm101 derm101: Similar to Dermnet NZ, this resource provides images within the dermatology atlas. It has different subscription plans to use the image library.

  12. Derm7pt Kawahara et al. (2018): This dataset has around 2,000 dermoscopic and clinical images of skin lesions, with a 7-point check-list criteria.

  13. The Cancer Genome Atlas 45: This dataset is one of the largest collections of pathological skin lesion slides with 793 cases. It is publicly available for anyone in the research community to use.

(a) (b) (c) (d)
(e) (f) (g) (h)
Figure 1: Illustration of different types of dermoscopic skin lesion where (a) Nevi (b) Melanoma (c) BCC (d) Actinic Keratosis (e) Benign Keratosis (f) Dermatofibroma (g) Vascular Lesion (h) SCC Tschandl et al. (2018)

2.2 Artificial Intelligence in Dermoscopic Images

Dermoscopy is the inspection/examination of skin lesions with a dermatoscope device consisting of a high-quality magnifying lens and a (polarizable) illumination system. Dermoscopic images are captured with high-resolution digital single-lens reflex (DSLR) or smartphone camera attachments. The use of dermoscopic images for AI algorithms is becoming a very popular research field since the introduction of many large publicly available dermoscopic datasets consisting of different types of benign and cancerous skin lesions, as shown in Fig. 1. There have been multiple AI studies on lesion diagnosis using dermoscopic skin lesion datasets, which are listed below.

  1. Codella et al. Codella et al. (2017) developed an ensemble of deep learning algorithms on the ISIC-2016 dataset and compared the performance of this network with 8 dermatologists for the classification of 100 skin lesions as benign or malignant. The ensemble method outperformed the average performance of dermatologists by achieving an accuracy of 76% and specificity of 62% versus 70.5% and 59% achieved by dermatologists.

  2. Haenssle et al. Haenssle et al. (2018) trained a deep learning method InceptionV4 on a large dermoscopic dataset consisting of more than 100,000 benign lesions and melanoma images and compared the performance of a deep learning method with 58 dermatologists. On the test set of 100 cases (75 benign lesions and 25 melanoma cases), dermatologists had an average sensitivity of 86.6% and specificity of 71.3%, while the deep learning method achieved a sensitivity of 95% and specificity of 63.8%.

  3. Brinker et al. Brinker et al. (2019b) compared the performance of 157 board-certified dermatologists at 12 German university hospitals with a deep learning method (ResNet50) for 100 dermoscopic images (MClass-D) consisting of 80 nevi and 20 melanoma cases. Dermatologists achieved an overall sensitivity of 74.1%, specificity of 60.0% and an AUROC of 0.67 on the dermoscopic dataset whereas a deep learning method achieved a specificity of 69.2% and a sensitivity of 84.2%.

  4. Tschandl et al. Tschandl et al. (2019) used popular deep learning architectures known as InceptionV3 and ResNet50 on a combined dataset of 7,895 dermoscopic and 5,829 close-up lesion images for diagnosis of non-pigmented skin cancers. The performance is compared with 95 dermatologists divided into three groups based on experience. The deep learning algorithms achieved accuracy on par with human experts and exceeded the human groups with beginner and intermediate raters.

2.3 Artificial Intelligence in Clinical Images

Clinical images are routinely captured of different skin lesions with mobile cameras for remote examination and incorporation into patient medical records, as shown in Fig. 2.. Since clinical images are captured with different cameras with variable backgrounds, illuminance and color, these images provide different insights for dermoscopic images.

(a) (b) (c) (d)
Figure 2: Illustration of different types of clinical skin lesion where (a) Benign Keratosis (b) Melanoma (c) BCC (d) SCC Yang et al. (2018)
  1. Yang et al. Yang et al. (2018) performed clinical skin lesion diagnosis using representation inspired by the ABCD rule on the SD-198 dataset. They compared the performance of the proposed methods with deep learning methods and dermatologists. It achieved a score of 57.62% (accuracy) in comparison to the best performing deep learning method (ResNet), which achieved 53.35%. When compared to the clinicians, only senior clinicians who have considerable experience in skin disease achieved an average accuracy of 83.29%.

  2. Han et al. Han et al. (2018a)

    trained a deep learning architecture (ResNet-152) to classify the clinical images of 12 skin diseases on an Asan training dataset, a MED-NODE dataset, and atlas site images, and tested it on an Asan testing set and an Edinburgh Dataset (Dermofit). The algorithm’s performance was on par with the team of 16 dermatologists on 480 randomly chosen images from the Asan test dataset (260 images) and the Edinburgh dataset (220 images), whereas the AI system outperformed dermatologists in the diagnosis of BCC.

  3. Fujisawa et al. Fujisawa et al. (2019) tested a deep learning method on 6,009 clinical images of 14 diagnoses, including both malignant and benign conditions. The deep learning algorithm achieved a diagnostic accuracy of 76.5% which is superior to the performance of 13 board-certified dermatologists (59.7%) and nine dermatology trainees (41.7%) on a 140-image dataset.

  4. Brinker et al. Brinker et al. (2019a) compared the performance of 145 dermatologists and a deep learning method (ResNet50) for the test case of 100 clinical skin lesion images (MClass-ND) consisting of 80 nevi cases and 20 biopsy-verified melanoma cases. The dermatologists achieved an overall sensitivity of 89.4%, specificity of 64.4% and an AUROC of 0.769 whereas a deep learning method achieved a same sensitivity score and a specificity of 69.2%.

(a) (b) (c) (d)
Figure 3: Illustration of different types of histopathology images where (a) Nevi (b) Melanoma (c) BCC (d) SCC 14

2.4 Artificial Intelligence in Histopathology Images

The diagnosis of skin cancer is confirmed by dermatopathologists based on microscopic evaluation of a tissue biopsy. Deep learning solutions have been successful in the field of digital pathology with whole-slide imaging. Examples of histopathology images of skin lesions are shown in Fig. 3. These techniques are used for the classification of biopsy tissue specimens to diagnose the number of cancers such as skin, lung, and breast. In this section, we explore the deep learning methods used in digital histopathology specific to skin cancer.

  1. Hekler et al. Hekler et al. (2019) used a deep learning method (ResNet50) to compare the performance of pathologists in classifying melanoma and nevi. The deep learning model was trained on a dataset of 595 histopathology images (300 melanoma and 295 nevi) and tested on 100 images (melanoma/nevi = 1:1). The total discordance with the histopathologist was 18% for melanoma, 20% for nevi, and 19% for the full set of images.

  2. Jiang et al. Jiang et al. (2019) proposed the use of a deep learning algorithm on smartphone-captured digital histopathology images (MOI) for the detection of BCC. They found that the performance of the algorithm on MOI and Whole Slide Imaging (WSI) is comparable with 0.95 AUC score. They introduced a deep segmentation network for in-depth analysis of the hard cases to further improve the performance with 0.987 AUC, 0.97 sensitivity, 0.94 specificity score.

  3. Cruz-Roa et al. Cruz-Roa et al. (2013)

    used a deep learning architecture to discriminate between BCC and normal tissue patterns on 1,417 images from 308 Region of Interests (ROI) of skin histopathology images. They compared the deep learning method with traditional machine learning with feature descriptors, including the bag of features, canonical and Haar-based wavelet transform. The deep learning architecture proved superior over the traditional approaches by achieving 89.4% in F-Measure and 91.4% in balanced accuracy.

  4. Xie et al. Xie et al. (2019) introduced a large dataset of 2,241 histopathological images of 1,321 patients from 2008 to 2018. They used two deep learning architectures, VGG19 and ResNet50, on the 9.95 million patches generated on 2,241 histopathological images to test the classification of melanoma and nevi on different magnification scales. They achieved high accuracy in distinguishing melanoma from nevi with average F1 (0.89), Sensitivity (0.92), Specificity (0.94) and AUC (0.98).

3 Challenges in Artificial Intelligence

With deep learning algorithms surpassing the benchmarks of popular computer vision datasets in a short period, the same trend could be expected in the skin lesion diagnosis challenge as well. However, as we further explore the skin lesion diagnosis challenge, this task appear to be not straightforward like ImageNet, PASCAL-VOC, MS-COCO challenges in a non-medical domain

Krizhevsky et al. (2012); Everingham et al. (2010); Lin et al. (2014). There are intra-class similarities and inter-class dissimilarities regarding color, texture, size, place, and appearance in the visual appearance of skin lesions. Deep learning algorithms generally require a substantial amount of diverse, balanced, and high-quality training data that represent each class of skin lesions to improve diagnostic accuracy. For skin lesion datasets of various modalities, there are many more issues related to the diagnosis of skin cancer with AI solutions as discussed below.

3.1 Performance of Deep learning and Unbalanced Datasets

The performance of deep learning algorithms mostly depends on the quality of image datasets rather than tuning the hyper-parameters of networks, as is commonly seen in the different publicly available skin lesion datasets. There are generally more cases of benign skin lesions rather than malignant lesions. Most of the deep learning architectures are designed on a balanced dataset, such as ImageNet, which consists of 1,000 images per class (1000 classes) Krizhevsky et al. (2012)

. Hence, the performance of a deep learning algorithm usually suffers from unbalanced datasets, despite using tuning tricks like a penalty for false-negatives found in a minor skin lesion class during training using custom loss functions.

3.2 Patients’ Medical History and Clinical Meta-data

Patients’ medical history, social habits, and clinical meta-data are considered when making a skin cancer diagnosis. It is very important to know the diagnostic meta-data, such as patient and family history of skin cancer, age, ethnicity, sex, general anatomic site, size and structure of the skin lesion, while performing a visual inspection of a suspected skin lesion with dermoscopy. Hence, only image-based deep learning algorithms used for the diagnosis of skin cancer falter on key aspects of patient and clinical information. It is proven in a previous study Haenssle et al. (2018) that both ‘beginners’ and ‘skilled’ dermatologists’ performance is improved with the availability of clinical information and that they performed better than deep learning algorithms. Unfortunately, both patient history and clinical meta-data are missing in the most publicly available skin lesion datasets.

3.3 ABCDE Rule and Time-line Datasets

In the clinical setting, a suspicious lesion is visually inspected with the help of dermoscopy. The ABCDE rule is considered an important rule for differentiating benign moles (nevi) from melanoma. This includes whether the lesion is asymmetrical, has irregular borders, displays multiple colors, whether the diameter of the lesion is greater than six millimetres, and if there has been any evolution or change in the composition of the lesion. Despite the availability of dermoscopic datasets of skin lesions, deep learning algorithms do not work the same way or look for a pattern similar to the ABCDE rule trusted by clinicians. It is mainly due to the complexity of pattern recognition for the characteristics of skin cancers in medical imaging. That is why, despite recent attempts by researchers to demystify the working of deep learning algorithms, such efforts are still considered as a black-box approach, especially in medical imaging. Since there are no timeline dermoscopic datasets available publicly, it is not possible to determine the change of a lesion’s characteristics according to the evolution of the ABCDE rule.

3.4 Biopsy is a Must

Even if skin cancer is confirmed by AI solutions with a high confidence rate, a biopsy and histological test must still be undertaken to confirm a diagnosis. The diagnostic accuracy of deep learning algorithms could be misleading as well. For example, if a testing set consists of 20 melanoma and 80 nevi cases, and the overall diagnostic accuracy is 90% (100% in nevi and 50% in melanoma cases), it is dangerous for a deep learning algorithm to be used in this case as a means to deliver a diagnosis of melanoma. As misdiagnosis of a cancer patient by a deep learning algorithm could risk a fatality, a biopsy should be taken to ensure safety and confirm the algorithm’s diagnosis.

3.5 Inter-class Similarities (Mimics of Skin Lesions)

A number of skin lesions can mimic skin cancer in both clinical and microscopic settings, which could result in misdiagnosis. For example, in clinical and dermoscopic images, seborrheic keratosis can mimic skin cancers including BCC, SCC, and melanoma. In histopathology images, there are many histologic mimics of BCC such as SCC, benign follicular tumors, basaloid follicular hamartoma, a tumor of follicular infundibulum, syringoma, and microcystic adnexal carcinoma Stanoszek et al. (2017). Hence, deep learning algorithms, when trained on limited classes of skin lesions in a dataset, do not reliably distinguish skin cancers from their known mimics.

3.6 Intra-class Dissimilarities

Several skin lesions have intra-class dissimilarities in terms of color, attribute, texture, size, site. Hence, these skin lesions are further categorized into many sub-categories based on visual appearance. For example, the color of most melanomas is black because of the dark pigment of melanin. But certain melanomas are found to be of normal skin color, reddish, and pinkish looking. Similarly, BCC has many subcategories, such as nodular BCC, superficial BCC, morphoeic BCC, basosquamous BCC, and their appearance is completely different from each other, ranging from white to red in color as shown in Fig. 4.

(a) (b) (c) (d)
Figure 4: Illustration of intra-class dissimilarities in BCC (a) Nodular BCC (b) Superficial BCC (c) Morphoeic BCC (d) Basosquamous BCC 14

3.7 Communication Barrier between AI and Dermatologists

Sometimes even experts in computer vision find it hard to understand the decisions made by deep learning frameworks. For example, if there is an algorithm that is 85% accurate for the diagnosis of skin cancer, it is often very difficult to understand why the algorithm is making wrong inferences on the rest of the 15% of cases, and how to improve those decisions. These algorithms are not usually similar or representative of the ways in which clinicians make such decisions. Hence, deep learning algorithms are often deemed as a black box solution that does not offer clear explanation for its conclusion, and often they only provide an output in confidence probability ranging from 0 to 1 for classification of each skin lesion in a test set. Currently, it is not clear how dermatologists would interpret deep learning models’ outcomes in diagnosing of skin cancer.

3.8 Noisy Real-life Data with Heterogeneous Data Sources

In the current datasets of skin lesions, the dermoscopic images are captured with high-resolution DSLR cameras and in an optimal environment of lighting and distance of capture. A deep learning algorithm trained on these high-quality dermoscopic datasets, achieving a reasonable diagnostic accuracy, could potentially be scaled to smart-phone vision applications. When this model is tested on multiple smart phone captured images by different cameras in different lighting conditions and distances, the same diagnostic accuracy is hard to achieve. Deep learning algorithms are found to be highly sensitive to which camera devices are used to capture the data, and their performance degrades if a different type of camera device is used for testing. Patient-provided self-captured skin images are frequently of low-quality and are not suitable for digital dermatology Weingast et al. (2013); Hogan et al. (2015).

3.9 Race, Ethnicity and Population

Most of the cases in the current skin lesion datasets belong to fair-skinned individuals rather than brown or dark-skinned persons. Although the risk of developing skin cancer is indeed relatively high among the fair-skinned person population, people with dark skin can also develop skin cancer and are frequently diagnosed at later stages Hu et al. (2006). Skin cancer represents 4 to 5%, 2 to 4%, 1 to 2% of all cancers in Hispanics, Asians, and Blacks, respectively Gloster Jr and Neal (2006). Hence, deep learning frameworks validated for the diagnosis of skin cancer in fair-skinned people has a greater risk of misdiagnosing those with darker skin Marcus and Davis (2019). In a recent study, Han et al. Han et al. (2018a) trained a deep learning algorithm on an Asan training dataset consisting of skin lesions from Asians. They reported an accuracy of 81% on the Asian testing set, whereas they reported an accuracy of only 56% on the Dermofit dataset, which consists of skin lesions of Caucasian people. Therefore, this drop-in accuracy signifies a lack of transferability of the learned features of deep learning algorithms across datasets that contain persons of a different race, ethnicity, or population.

3.10 Rare Skin Cancer and Other Skin Conditions

BCC, SCC and melanoma collectively comprise 98% of all skin cancers. However, there are other skin cancers, including Merkel cell carcinoma (MCC), appendageal carcinomas, cutaneous lymphoma, sarcoma, kaposi sarcoma, and cutaneous secondaries, that are ignored by most algorithms. Beside these rare skin cancers, there are certain other skin conditions, such as ulcers, skin infections, neoplasms, and non-infectious granulomas, that could mimic skin lesions. If deep learning algorithms are trained on datasets that do not have adequate cases of these rare skin cancers and other mentioned skin conditions, there is a high risk of misdiagnosis when it is tested on these skin conditions.

3.11 Incomplete Diagnosis Pipeline for Artificial Intelligence

In the clinical setting, the diagnosis of skin cancer is made by inspecting the skin lesion with or without dermoscopy, followed by confirmatory biopsy and pathological examination. The major issue with the current publicly available skin lesion datasets is that they lack complete labels related to the diagnosis performed by a dermatologist. Nevertheless, most of the classification labels for dermoscopic skin lesion images are determined by pathological examination. Still, these dermoscopic and clinical skin lesion datasets do not have corresponding pathological classification labels to develop a complete diagnosis pipeline for AI.

4 Opportunities

AI researchers invariably claim their systems exceed the performance of dermatologists for the diagnosis of skin cancer. But this picture is far from reality, as these experiments are performed in closed systems with a defined set of rules. With the many challenges mentioned in the above section, the nature of these reported performance evaluations is nowhere near the real-life diagnosis performed by clinicians treating skin cancer. Often, deep learning algorithms are deemed as opaque, as they only learn from pixel values of imaging datasets and do not have any domain knowledge or perform logical inferences to establish the relationship between different types of skin lesions Marcus and Davis (2019). But, in the future, deep learning could do very well for the diagnosis of skin cancer with the given opportunities listed below.

4.1 Balanced Dataset and Selection of Cases

A balanced dataset is critical for the good performance of deep learning algorithms used for classification tasks. Hence, balanced datasets are required with a selection of cases that completely represent the category of that particular skin lesion, and the input of experienced dermatologists could be very helpful for this selection.

4.2 Diverse Datasets

Deep learning networks are often criticised for social biases due to most of the imaging data belonging to fair-skinned persons. Skin lesion datasets need to have racial diversity, i.e., they must add equally distributed skin lesion cases from fair-skinned and dark-skinned people to reduce social or ethnic bias in deep learning models. The same concern can be extended to age, especially when the degree of skin aging or surrounding solar damage can influence the dataset and decision-making.

4.3 Data-Augmentation

Data augmentation techniques may mitigate many limitations of datasets, such as unbalanced data among the classes of skin lesions and heterogeneous sources of data, by adding augmented samples with different image transformations, such as rotation, random crop, horizontal and vertical flip, translation, shear, color jitter, and colorspace. It is proven in many studies that data augmentation improved the diagnosis of skin cancer Vasconcelos and Vasconcelos (2017); Pham et al. (2018). In the HAM10000 dataset Tschandl et al. (2018), the skin lesion images were captured at different magnifications or angles or with different cameras, a process known as natural data augmentation. Notably, Goyal et al. Goyal and Yap (2018) used deep learning architecture called Faster R-CNN to develop the algorithm to generate augmented copies similar to the natural data-augmentation for other skin lesion datasets.

4.4 Generative Adversarial Networks

Generative adversarial networks (GAN) are recently developed deep learning architectures that are attracting interest in the medical imaging community. GAN is mainly used to generate high-quality fake imaging data to overcome a limited dataset Goodfellow et al. (2014); Yi et al. (2019); Wei et al. (2019b). For skin cancer, GAN can be used to generate realistic synthetic skin lesion images to overcome the lack of annotated data Bissoto et al. (2018)

. The distribution of skin lesions in publicly available datasets is heavily skewed by each class’s prevalence among patients, and GAN can be used to generate imaging data for under-represented skin lesion classes or rare classes of skin cancer, such as MCC, sebaceous carcinoma, or kaposi sarcoma.

4.5 Identifying Sub-categories

There could be many visual intra-class dissimilarities in the appearance of skin lesions in terms of texture, color, and size. In most of the publicly available datasets, the collection of skin lesions belongs to each superclass rather than dividing them into sub-categories. Dealing with many intra-class dissimilarities and inter-class similarities (mimics of a skin lesion) in the skin lesion dataset, it is challenging for deep learning algorithms to classify or differentiate such lesions. As a possible solution to deal with this issue, sub-categories of each skin lesion should be treated as different classes in the dataset used for training deep learning algorithms. However, this will require a greater volume of training images and it will also be more challenging to translate into clinical practice. Therefore, subcategorization would require a certain degree of suspicion or a reasonable pre-test probability to adequately aid the clinician choosing the algorithm.

4.6 Semantic Explanation of Prediction

To assist clinicians, deep learning algorithms need to provide a semantic explanation rather than just a confidence score for the prediction of skin lesions. One possible solution could be for deep learning algorithms using longitudinal datasets to provide a semantic explanation in terms of the ABCDE rule.

4.7 Multiple Models for Diagnosis of Skin Cancer

Rather than relying on a single AI solution for the diagnosis of skin cancer, multiple deep learning models can evaluate different features or aspects of skin lesions, submit predictions, and generate a final conclusion. In this regard, cloud computational power and storage is becoming more affordable and it will be possible to host multiple models to assist dermatologists in the diagnosis of skin cancer, around the world in parallel (or in synchrony).

4.8 Combining Clinical Information and Deep Features

Clinical meta-data and patient history are considered clinically important in the diagnosis of skin cancer. This information can provide insight beyond the imaging features used by deep learning algorithms. Hence, there is a need to develop data fusion algorithms that can combine features comprised of clinical information with imaging features from deep learning models to provide final predictions of the diagnosis of skin cancer. In a recent study, Pacheco et al. Pacheco and Krohling (2019) combined deep learning models (clinical images) and patient clinical information to achieve approximately 7% improvement in the balanced prediction accuracy.

4.9 Multi-modality Solution: Complete Diagnosis Pipeline

If corresponding histopathological data for dermoscopic skin lesions were available in datasets, we could develop a complete AI solution similar to a dermatologist’s diagnosis pipeline. In the first step, an AI solution is used to classify dermoscopic skin lesions, with a deep learning algorithm trained on the dermoscopic dataset. For suspected cases, the deep learning algorithm can be developed on a pathological dataset to determine whether the lesion is cancerous or not.

4.10 Rigorous Clinical Validation

It is a well-known fact, for both clinicians and AI researchers, that mistakes can inform future decision-making. Since we cannot afford misdiagnosis by technology, it is better to keep AI solutions in the background for rigorous validation of noisy data coming from real patients and for improving the predictions of these technological systems to date, until they are finally validated to provide useful insights into the diagnosis of skin cancer and assist clinicians either in hospital and remote settings.

5 Conclusion

Research involving AI is making encouraging progress in the diagnosis of skin cancer. Despite the various claims of deep learning algorithms surpassing clinicians’ performance in the diagnosis of skin cancer, there are far more challenges faced by these algorithms to become a complete diagnostic system. Because such experiments are performed in controlled settings, algorithms are never tested in the real-life diagnosis of skin cancer patients. The real-world diagnosis process requires taking into account a patient’s ethnicity, skin, hair and eye color, occupation, illness, medicines, existing sun damage, the number of nevi, and lifestyle habits (such as sun exposure, smoking, and alcohol intake), clinical history, the respond to previous treatments, and other information from the patient’s medical records. However, current deep learning models predominantly rely on only patients’ imaging data. Moreover, such systems often risk a misdiagnosis whenever they are applied to skin lesions or conditions that are not present in the training dataset. This paper further explores opportunities to build robust algorithms to assist clinicians in the diagnosis of skin cancer. Computer vision and dermatologist societies need to work together to improve current AI solutions and enhance the diagnostic accuracy of methods used for the diagnosis of skin cancer. AI has the potential to deliver a paradigm shift in the diagnosis of skin cancer, and thus a cost-effective, remotely accessible, and accurate healthcare solution.

Search strategy and selection criteria

We used Google Scholar and PubMed to find relevant manuscripts. We restricted our search to papers published in English between Jan 1, 2012, and Nov 15, 2019. We used the following terms in different combinations: “skin cancer”, ”skin lesions” “deep learning”, “artificial Intelligence”, “dermatologists”, “clinical images”, “dermoscopic”, ”histopathology” “artificial Intelligence and skin cancer”, “skin cancer and deep learning”, “dermatologists and deep learning”, “skin cancer datasets”, “lesion diagnosis and deep learning”, “clinical information, deep learning and skin cancer”, “data augmentation and skin cancer”,“GAN and skin cancer”.


  • Z. Apalla, A. Lallas, E. Sotiriou, E. Lazaridou, and D. Ioannides (2017) Epidemiological trends in skin cancer. Dermatology practical & conceptual 7 (2), pp. 1. Cited by: §1.
  • G. Argenziano, H. Soyer, V. De Giorgi, D. Piccolo, P. Carli, and M. Delfino (2000) Interactive atlas of dermoscopy (book and cd-rom). Cited by: item 2.
  • Y. Bengio, I. Goodfellow, and A. Courville (2017) Deep learning. Vol. 1, Citeseer. Cited by: §1.
  • A. Bissoto, F. Perez, E. Valle, and S. Avila (2018) Skin lesion synthesis with generative adversarial networks. In OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis, pp. 294–302. Cited by: §4.4.
  • F. Bray, J. Ferlay, I. Soerjomataram, R. L. Siegel, L. A. Torre, and A. Jemal (2018) Global cancer statistics 2018: globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: a cancer journal for clinicians 68 (6), pp. 394–424. Cited by: §1.
  • T. J. Brinker, A. Hekler, A. H. Enk, J. Klode, A. Hauschild, C. Berking, B. Schilling, S. Haferkamp, D. Schadendorf, S. Fröhling, et al. (2019a)

    A convolutional neural network trained with dermoscopic images performed on par with 145 dermatologists in a clinical melanoma image classification task

    European Journal of Cancer 111, pp. 148–154. Cited by: item 4.
  • T. J. Brinker, A. Hekler, A. H. Enk, J. Klode, A. Hauschild, C. Berking, B. Schilling, S. Haferkamp, D. Schadendorf, T. Holland-Letz, et al. (2019b) Deep learning outperformed 136 of 157 dermatologists in a head-to-head dermoscopic melanoma image classification task. European Journal of Cancer 113, pp. 47–54. Cited by: item 3.
  • T. Ching, D. S. Himmelstein, B. K. Beaulieu-Jones, A. A. Kalinin, B. T. Do, G. P. Way, E. Ferrero, P. Agapow, M. Zietz, M. M. Hoffman, et al. (2018) Opportunities and obstacles for deep learning in biology and medicine. Journal of The Royal Society Interface 15 (141), pp. 20170387. Cited by: §1.
  • N. C. Codella, D. Gutman, M. E. Celebi, B. Helba, M. A. Marchetti, S. W. Dusza, A. Kalloo, K. Liopyris, N. Mishra, H. Kittler, et al. (2018) Skin lesion analysis toward melanoma detection: a challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In Biomedical Imaging (ISBI 2018), 2018 IEEE 15th International Symposium on, pp. 168–172. Cited by: item 1.
  • N. C. Codella, Q. Nguyen, S. Pankanti, D. A. Gutman, B. Helba, A. C. Halpern, and J. R. Smith (2017) Deep learning ensembles for melanoma recognition in dermoscopy images. IBM Journal of Research and Development 61 (4/5), pp. 5–1. Cited by: item 1.
  • M. Combalia, N. C. Codella, V. Rotemberg, B. Helba, V. Vilaplana, O. Reiter, A. C. Halpern, S. Puig, and J. Malvehy (2019) BCN20000: dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288. Cited by: item 1.
  • A. A. Cruz-Roa, J. E. A. Ovalle, A. Madabhushi, and F. A. G. Osorio (2013) A deep learning architecture for image representation, visual interpretability and automated basal-cell carcinoma cancer detection. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 403–410. Cited by: item 3.
  • H. A. Daanen and F. B. Ter Haar (2013) 3D whole body scanners revisited. Displays 34 (4), pp. 270–275. Cited by: §1.
  • [14] DermNet nz. External Links: Link Cited by: Figure 3, item 10, Figure 4.
  • A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542 (7639), pp. 115–118. Cited by: §1, §2.
  • M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman (2010) The pascal visual object classes (voc) challenge. International journal of computer vision 88 (2), pp. 303–338. Cited by: §3.
  • S. C. Foundation (2017) Skin cancer facts and statistics. Note: Online External Links: Link Cited by: §1.
  • Y. Fujisawa, Y. Otomo, Y. Ogata, Y. Nakamura, R. Fujita, Y. Ishitsuka, R. Watanabe, N. Okiyama, K. Ohara, and M. Fujimoto (2019) Deep-learning-based, computer-aided classifier developed with a small dataset of clinical images surpasses board-certified dermatologists in skin tumour diagnosis. British Journal of Dermatology 180 (2), pp. 373–381. Cited by: item 3.
  • I. Giotis, N. Molders, S. Land, M. Biehl, M. F. Jonkman, and N. Petkov (2015) MED-node: a computer-assisted melanoma diagnosis system using non-dermoscopic images. Expert systems with applications 42 (19), pp. 6578–6585. Cited by: item 5.
  • H. M. Gloster Jr and K. Neal (2006) Skin cancer in skin of color. Journal of the American Academy of Dermatology 55 (5), pp. 741–760. Cited by: §3.9.
  • I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio (2014) Generative adversarial nets. In Advances in neural information processing systems, pp. 2672–2680. Cited by: §4.4.
  • M. Goyal, N. Reeves, S. Rajbhandari, and M. H. Yap (2018) Robust methods for real-time diabetic foot ulcer detection and localization on mobile devices. IEEE journal of biomedical and health informatics. Cited by: §1.
  • M. Goyal and M. H. Yap (2018) Region of interest detection in dermoscopic images for natural data-augmentation. arXiv preprint arXiv:1807.10711. Cited by: §4.3.
  • [24] M. Goyal Artificial intelligence in dermatology. DermNet NZ. External Links: Link Cited by: §1.
  • H. A. Haenssle, C. Fink, R. Schneiderbauer, F. Toberer, T. Buhl, A. Blum, A. Kalloo, A. B. H. Hassen, L. Thomas, A. Enk, et al. (2018) Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Annals of Oncology 29 (8), pp. 1836–1842. Cited by: item 2, §3.2.
  • S. S. Han, M. S. Kim, W. Lim, G. H. Park, I. Park, and S. E. Chang (2018a) Classification of the clinical images for benign and malignant cutaneous tumors using a deep learning algorithm. Journal of Investigative Dermatology 138 (7), pp. 1529–1538. Cited by: item 6, item 7, item 2, §3.9.
  • S. S. Han, G. H. Park, W. Lim, M. S. Kim, J. Im Na, I. Park, and S. E. Chang (2018b) Deep neural networks show an equivalent and often superior performance to dermatologists in onychomycosis diagnosis: automatic construction of onychomycosis datasets by region-based convolutional deep neural network. PloS one 13 (1), pp. e0191493. Cited by: item 6.
  • M. Havaei, A. Davy, D. Warde-Farley, A. Biard, A. Courville, Y. Bengio, C. Pal, P. Jodoin, and H. Larochelle (2017) Brain tumor segmentation with deep neural networks. Medical image analysis 35, pp. 18–31. Cited by: §1.
  • A. Hekler, J. S. Utikal, A. H. Enk, C. Berking, J. Klode, D. Schadendorf, P. Jansen, C. Franklin, T. Holland-Letz, D. Krahl, et al. (2019) Pathologist-level classification of histopathological melanoma images with deep neural networks. European Journal of Cancer 115, pp. 79–83. Cited by: item 1.
  • K. Hogan, J. Cullan, V. Patel, A. Rajpara, and D. Aires (2015) Overcalling a teledermatology selfie: a new twist in a growing field. Dermatology online journal 21 (6). Cited by: §3.8.
  • A. Hosny, C. Parmar, J. Quackenbush, L. H. Schwartz, and H. J. Aerts (2018) Artificial intelligence in radiology. Nature Reviews Cancer 18 (8), pp. 500–510. Cited by: §1.
  • S. Hu, R. M. Soza-Vento, D. F. Parker, and R. S. Kirsner (2006) Comparison of stage at diagnosis of melanoma among hispanic, black, and white patients in miami-dade county, florida. Archives of Dermatology 142 (6), pp. 704–708. Cited by: §3.9.
  • Y. Jiang, J. Xiong, H. Li, X. Yang, W. Yu, M. Gao, X. Zhao, Y. Ma, W. Zhang, Y. Guan, et al. (2019) Recognizing basal cell carcinoma on smartphone-captured digital histopathology images with deep neural network. British Journal of Dermatology. Cited by: item 2.
  • J. Kawahara, S. Daneshvar, G. Argenziano, and G. Hamarneh (2018) Seven-point checklist and skin lesion classification using multitask multimodal neural nets. IEEE journal of biomedical and health informatics 23 (2), pp. 538–546. Cited by: item 12.
  • A. Krizhevsky, I. Sutskever, and G. E. Hinton (2012) Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105. Cited by: §3.1, §3.
  • T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014) Microsoft coco: common objects in context. In European conference on computer vision, pp. 740–755. Cited by: §3.
  • G. Marcus and E. Davis (2019) Rebooting ai: building artificial intelligence we can trust. Pantheon. Cited by: §3.9, §4.
  • T. Mendonça, P. M. Ferreira, J. S. Marques, A. R. Marcal, and J. Rozeira (2013) PH 2-a dermoscopic image database for research and benchmarking. In Engineering in Medicine and Biology Society (EMBC), 2013 35th Annual International Conference of the IEEE, pp. 5437–5440. Cited by: item 4.
  • A. G. Pacheco and R. A. Krohling (2019) The impact of patient clinical information on automated skin cancer detection. arXiv preprint arXiv:1909.12912. Cited by: §4.8.
  • T. Pham, C. Luong, M. Visani, and V. Hoang (2018) Deep cnn and data augmentation for skin lesion classification. In Asian Conference on Intelligent Information and Database Systems, pp. 573–582. Cited by: §4.3.
  • P. Rajpurkar, J. Irvin, K. Zhu, B. Yang, H. Mehta, T. Duan, D. Ding, A. Bagul, C. Langlotz, K. Shpanskaya, et al. (2017) Chexnet: radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225. Cited by: §1.
  • H. W. Rogers, M. A. Weinstock, S. R. Feldman, and B. M. Coldiron (2015) Incidence estimate of nonmelanoma skin cancer (keratinocyte carcinomas) in the us population, 2012. JAMA dermatology 151 (10), pp. 1081–1086. Cited by: §1.
  • L. M. Stanoszek, G. Y. Wang, and P. W. Harms (2017) Histologic mimics of basal cell carcinoma. Archives of pathology & laboratory medicine 141 (11), pp. 1490–1502. Cited by: §3.5.
  • W. Street (2019) Cancer facts & figures 2019. American Cancer Society: Atlanta, GA, USA. Cited by: §1.
  • [45] The cancer genome atlas program. External Links: Link Cited by: item 13.
  • N. Tomita, B. Abdollahi, J. Wei, B. Ren, A. Suriawinata, and S. Hassanpour (2019) Attention-based deep neural networks for detection of cancerous and precancerous esophagus tissue on histopathological slides. JAMA network open 2 (11), pp. e1914645–e1914645. Cited by: §1.
  • P. Tschandl, C. Rosendahl, B. N. Akay, G. Argenziano, A. Blum, R. P. Braun, H. Cabo, J. Gourhant, J. Kreusch, A. Lallas, et al. (2019) Expert-level diagnosis of nonpigmented skin cancer by combined convolutional neural networks. JAMA dermatology 155 (1), pp. 58–65. Cited by: item 4.
  • P. Tschandl, C. Rosendahl, and H. Kittler (2018) The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci. Data 5, pp. 180161. External Links: Document Cited by: Figure 1, item 1, §4.3.
  • C. N. Vasconcelos and B. N. Vasconcelos (2017) Convolutional neural network committees for melanoma classification with classical and expert knowledge based image transforms data augmentation. arXiv preprint arXiv:1702.07025. Cited by: §4.3.
  • S. Wang, Z. Su, L. Ying, X. Peng, S. Zhu, F. Liang, D. Feng, and D. Liang (2016) Accelerating magnetic resonance imaging via deep learning. In 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), pp. 514–517. Cited by: §1.
  • J. W. Wei, L. J. Tafe, Y. A. Linnik, L. J. Vaickus, N. Tomita, and S. Hassanpour (2019a) Pathologist-level classification of histologic patterns on resected lung adenocarcinoma slides with deep neural networks. Scientific reports 9 (1), pp. 3358. Cited by: §1.
  • J. Wei, A. Suriawinata, L. Vaickus, B. Ren, X. Liu, J. Wei, and S. Hassanpour (2019b) Generative image translation for data augmentation in colorectal histopathology images. In Proceedings of Machine Learning for Health Workshop at NeurIPS, Cited by: §4.4.
  • J. Weingast, C. Scheibböck, E. M. Wurm, E. Ranharter, S. Porkert, S. Dreiseitl, C. Posch, and M. Binder (2013) A prospective study of mobile phones for dermatology in a clinical setting. Journal of telemedicine and telecare 19 (4), pp. 213–218. Cited by: §3.8.
  • T. Würfl, F. C. Ghesu, V. Christlein, and A. Maier (2016) Deep learning computed tomography. In International conference on medical image computing and computer-assisted intervention, pp. 432–440. Cited by: §1.
  • P. Xie, K. Zuo, Y. Zhang, F. Li, M. Yin, and K. Lu (2019) Interpretable classification from skin cancer histology slides using deep learning: a retrospective multicenter study. arXiv preprint arXiv:1904.06156. Cited by: item 4.
  • J. Yang, X. Sun, J. Liang, and P. L. Rosin (2018) Clinical skin lesion diagnosis using representations inspired by dermatologist criteria. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1258–1266. Cited by: Figure 2, item 8, item 1.
  • J. Yang, X. Wu, J. Liang, X. Sun, M. Cheng, P. L. Rosin, and L. Wang (2019) Self-paced balance learning for clinical skin disease recognition. IEEE transactions on neural networks and learning systems. Cited by: item 9.
  • M. H. Yap, M. Goyal, F. Osman, E. Ahmad, R. Martí, E. Denton, A. Juette, and R. Zwiggelaar (2018a) End-to-end breast ultrasound lesions recognition with a deep learning approach. In Medical Imaging 2018: Biomedical Applications in Molecular, Structural, and Functional Imaging, Vol. 10578, pp. 1057819. Cited by: §1.
  • M. H. Yap, M. Goyal, F. M. Osman, R. Martí, E. Denton, A. Juette, and R. Zwiggelaar (2018b) Breast ultrasound lesions recognition: end-to-end deep learning approaches. Journal of Medical Imaging 6 (1), pp. 011007. Cited by: §1.
  • X. Yi, E. Walia, and P. Babyn (2019) Generative adversarial network in medical imaging: a review. Medical image analysis, pp. 101552. Cited by: §4.4.