Heterogeneity Loss to Handle Intersubject and Intrasubject Variability in Cancer

by   Shubham Goswami, et al.
IIIT Delhi

Developing nations lack adequate number of hospitals with modern equipment and skilled doctors. Hence, a significant proportion of these nations' population, particularly in rural areas, is not able to avail specialized and timely healthcare facilities. In recent years, deep learning (DL) models, a class of artificial intelligence (AI) methods, have shown impressive results in medical domain. These AI methods can provide immense support to developing nations as affordable healthcare solutions. This work is focused on one such application of blood cancer diagnosis. However, there are some challenges to DL models in cancer research because of the unavailability of a large data for adequate training and the difficulty of capturing heterogeneity in data at different levels ranging from acquisition characteristics, session, to subject-level (within subjects and across subjects). These challenges render DL models prone to overfitting and hence, models lack generalization on prospective subjects' data. In this work, we address these problems in the application of B-cell Acute Lymphoblastic Leukemia (B-ALL) diagnosis using deep learning. We propose heterogeneity loss that captures subject-level heterogeneity, thereby, forcing the neural network to learn subject-independent features. We also propose an unorthodox ensemble strategy that helps us in providing improved classification over models trained on 7-folds giving a weighted-F_1 score of 95.26 far, the best results on the C-NMC 2019 dataset for B-ALL classification.



There are no comments yet.


page 2


Cell nuclei classification in histopathological images using hybrid OLConvNet

Computer-aided histopathological image analysis for cancer detection is ...

Explainable Deep Learning in Healthcare: A Methodological Survey from an Attribution View

The increasing availability of large collections of electronic health re...

A nonparametric Bayesian approach to simultaneous subject and cell heterogeneity discovery for single cell RNA-seq data

The advent of the single cell sequencing era opens new avenues for the p...

Secure and Robust Machine Learning for Healthcare: A Survey

Recent years have witnessed widespread adoption of machine learning (ML)...

Automated Prostate Cancer Diagnosis Based on Gleason Grading Using Convolutional Neural Network

The Gleason grading system using histological images is the most powerfu...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Acute Lymphoblastic Leukemia (ALL) is a type of immature white blood cell cancer. As ALL progresses quickly in a few months, its early stage detection is crucial. In populous countries such as India, even a small percentage of people diagnosed with cancer can result in a large number of people requiring urgent attention and diagnosis in early stages of cancer. Artificial intelligence (AI) can be used to build affordable and easily deployable solution to classify cancer versus normal cells. This work focuses on the identification of cancer and healthy cells in B-lineage ALL cancer. This cancer constitutes approx. 20% of the pediatric cancers

manoharan2009cancer. The problem was explored earlier too (baseline-1; baseline-2), but the dataset were limited in size to less than 400 cell images. We have used C-NMC 2019 challenge dataset of IEEE ISBI 2019 that consists of nearly 14000 cell images (ref50). One of the earlier works on the C-NMC 2019 dataset was carried out by 10.1007/978-3-319-66179-7_50

, who presented stain deconvolutional layer based convolutional neural networks (CNN) model to classify healthy and cancer cells by projecting the image data to optical density space via stain deconvolution. However, a major limitation of the approach was that the train-test split was not done at the subject-level. Hence, the images of the same subject could be present in both the training and the test data. This can cause the classifier to fail on the prospective (new unseen) subjects’ data. In this paper, we conducted training and testing at the subject-level as described in the train-test splits of the C-NMC 2019 dataset, i.e., all images of a given subject will be present in either train or validation or test set. A number of works have been published on this dataset.

isbi3 used an ensemble of CNN and RNN, isbi4 used ensemble of Inception-v2, Inception-v3, and Densenet with cross entropy loss, isbi5 used an ensemble of Densenet, Resnet and VGG, isbi7 showed the comparison of different CNN architectures and used ResNeXt (50 and 101) as classifier with GAN to augment the data, while isbi8

used Resnet with Fischer vector aggregation and neighbourhood correction. Recently,

gehlot2020sdct introduced SDCT-AuxNet where the classifier uses features of CNN network and those from auxiliary classifier. Also, stain deconvolved quantity images are used instead of the traditional RGB images. However, none of these works handled intersubject or intrasubject heterogeneity in the dataset, failure of which can result in learning subject-dependent features by the classifier. In this paper, we have addressed the effect of subject-level heterogeneity.

2 Dataset Description

We have used C-NMC 2019 challenge dataset of IEEE ISBI 2019 that consists of segmented white blood cell images of 69 ALL cancer patients and 49 healthy subjects. This dataset is publicly available as C-NMC 2019 B-ALL classification challenge at The Cancer Imaging Archive (TCIA) (ref56; ref49) with the description of distribution of subjects into train and test splits. This dataset is prepared from microscopic images captured from the bone marrow aspirate smears of subjects. The images were normalized for stain color variability (ref53) and cells, marked by expert oncologists, were segmented (ref54). The dataset was collected by All India Institute of Medical Sciences (AIIMS) Delhi, India. This hospital receives some of the most critical cases from across the country, with many cases referred to AIIMS by other states’ regional hospitals. Hence, this dataset is extremely rich in terms of subject-level diversity that makes C-NMC 2019 dataset as one of the best available dataset for B-ALL classification.

3 Motivation

This problem of the discrimination of the healthy cells from cancer cells in B-ALL cancer is very challenging owing to the following reasons:

  1. Morphologically, the cells of the two classes (healthy and cancer) appear similar (Fig. 1).

    Figure 1: LEFT: Two cancer cells, RIGHT: Two healthy cells
  2. In biomedical applications, dataset invariably contain intersubject heterogeneity. Also, each subject’s data consists of a set of images that may contain intrasubject heterogeneity. These heterogeneities can force the classifier to learn subject-level and subject-specific features instead of learning class-specific features (Allison2014), leading to poor performance on the test set. This is to note that subject-level heterogeneity is not only limited to biomedical datasets, but is relevant wherever the dataset used for training consists of any subject-level bias, say, gender bias in datasets (bias). Also, C-NMC 2019 dataset suffers from a long tail distribution, i.e., there is a significant difference in the number of images across subjects. This can lead to overfitting at the subject-level and can force the classifier to give more weight to subjects with more images as compared to subjects with less number of images. This can lead to poor generalization on unseen subjects.

  3. There is a class imbalance in this dataset with almost double the number of images of the healthy class compared to the cancer class.

In order to deploy a reliable DL-based tool for B-ALL classification, we propose a deep learning based solution, addressing above challenges, in Sec-4

for capturing subject-level heterogeneity by: 1) data sampling strategy to capture subject-level heterogeneity in the training data; 2) new loss function, namely, heterogeneity loss for ALL cell classification that handles the fundamental problem of subject-level heterogeneity of data for better generalization of the model on unseen test data; and 3) by an unorthodox ensemble strategy that exploits confidence scores of independently trained Inception-v3 (trained on the proposed loss function) to arrive at the final decision.

4 Proposed methodology

4.1 Data Sampling

The data split provided in C-NMC 2019 consists of the training set of 101 subjects with a total of 12528 cell images and final test set of 17 subjects with a total of 2586 images. Test set labels are not provided and the final test scores are required to be checked via submission at the leaderboard (ref50). Due to the fewer number of subjects in the dataset, capturing of subject-level heterogeneity from the training data is hard. To address this problem, we sampled 7 different folds at the subject-level split from the training dataset in a stratified manner by maintaining the ratio of number of cancer cells to healthy cells across all the folds such that a subject’s data is either present in the training set or is present in the validation set in a given fold as shown in Table-1. Seven CNN networks are trained independently on each fold.

Data 1 2 3 4 5 6 7 Total
Cancer 1234 (13) 1166 (9) 1269 (9) 1275 (6) 1135 (9) 1197 (6) 1215 (8) 8491 (60)
Healthy 529 (9) 546 (7) 516 (6) 624 (4) 618 (6) 603 (4) 601 (5) 4037 (41)
Table 1: Cross-validation splits of training data into 7 folds: Number of cell images (number of subjects)

4.2 Network Architecture

We have used Inception-v3 deep convolution neural network (CNN) for solving this problem. Different kernel sizes (from as small as to large kernels) capture different receptive fields of the input image at a layer in CNN network (peng2017large). Inception-v3 CNN (szegedy2016rethinking)

with 42-layers and around 23 million parameters exploits this. It fulfils our requirement of the simultaneous use of different kernel sizes for capturing and exploiting information at different scales. We have initialized weights of Inception-v3 by pre-training on ImageNet dataset.

4.3 Proposed Heterogeneity Loss Function

The challenges discussed in Sec-3

have a striking resemblance to those encountered in face recognition domain.

10.1007/978-3-319-46478-7_31 handled intraclass variations in face recognition by applying a loss on the learned features along with the class centers. We hypothesize that not only images belonging to separate classes should be far apart in the feature space, images belonging to the same class should also form distinct and individual compact-feature clusters. Alternatively, contrastive loss (1640964; NIPS2014_5416) and triplet loss (triplet_cvpr)

were used on deep features to combat intraclass variations. However, these methods require image pairs or triplets, respectively, where the number of training pairs or triplets grow dramatically leading to an increase in computational complexity. This makes the training process very inconvenient

(10.1007/978-3-319-46478-7_31). On the other hand, we propose a new loss function, namely, heterogeneity loss, in equation (1) that uses multiple-instance centre loss to capture intraclass compactness along with the intersubject and intrasubject heterogeneity and does not suffer with the disadvantages of contrastive or triplet losses. The heterogeneity loss function is explained as follows. Let a mini-batch of size be defined as , where represents the deep feature of the sample in the mini-batch . Let and represent the number of classes and the number of subjects in the training set, respectively and, and denote the subject labels and class labels of the training set, respectively. We define two mappings: a) for all , represents the subject label of , and b) for all represents the class label over deep feature and subject label . We define mini-batch subsets of on class-level and subject-level as follows: for all , s.t. and for all , s.t. . represents the class centre for and represents the subject centre for . Next, we define the heterogeneity loss function as:


where , and are the hyper-parameters associated with . Centre loss was introduced by 10.1007/978-3-319-46478-7_31 to increase interclass distance and intraclass compactness to learn better discriminative features of the output class labels. consists of two components. The first component handles output class inference and consists of two losses: softmax-cross entropy loss shown in (2) and class centre loss shown in (3) that handles the inference. The second component tries to capture the heterogeneity at the subject-level for learning subject independent features by using centre loss as a multi-task auxiliary loss. This is achieved by the weighted sum of shown in (4) and shown in (5). This second component handles intersubject and intrasubject heterogeneity by geometrically forcing the output of the last fully connected layer because increases intrasubject compactness for each subject and hence, learns features common to all images belonging to the same subject. decreases intersubject distance between the subjects belonging to the same class. Increase in intra-subject compactness and decrease in inter-subject distance at the same time is important because it forces the network to learn subject-independent features. defined in (3) is the sum of mean centre loss with respect to each output class label over . The mean term for serves as class weights to handle class imbalance. Similarly, can handle subject imbalance for a given mini-batch.

5 Experiments and Results

5.1 Training

We used Pytorch along with NVIDIA GTX 1080 Ti for training. Single Inception-v3 model trains in around 4 hours. We used both Train time Augmentation (TrA) and Test time Augmentation (TTA). Initially, all the images were cropped to a fixed size of

pixels. For robust training, several augmentation techniques were applied such as HorizontalFlip, VerticalFlip, RandomRotation, Affine Transformation, adjustment of hue, saturation, contrast, brightness, and RandomCrop that resizes the images to for Inception-v3. For each fold we used two stage training of Inception-v3 model on loss with , and with values , and , respectively (tuned over multiple experiments), Adam optimizer and a batch size of 16. The first stage used a learning rate of and the best model was saved before the loss started diverging on the training set. In the second stage, we resumed training from the first phase’s checkpoint with . We also used Test time augmentation (TTA), called as 5-crop strategy with size crops: one each from the four corners and one from the centre of the unseen input cell image. Finally, average confidence scores on these 5 crops for each input image is calculated. Comparative performance of different loss configurations in Table-2 shows that heterogeneity loss indeed performs best. Fig-A1 shows the accuracy and convergence plots of loss on independently trained Inception-v3 model corresponding to each of the 7 folds. Fig-2 shows convergence of on subjects randomly sampled from the training and validation sets. Convergence of is important because is formulated using subject centers and also it shows that the classifier can handle class-subject imbalance. Overall convergence of would lead to a good generalization of the model on the unseen test data. Fig-2 also shows class-level T-SNE visualization (on the same random split as above) using the output of last layer of Inception-v3 as features.

Loss Function Accuracy (%) Weighted- (%)
84.66 84.25
86.14 85.84
88.88 88.62
90.05 89.94
Table 2: Performance of Inception-v3 with different loss configurations using training and validation sets provided on the challenge portal (ref50)

5.2 Ensemble-based Decision

One Inception-v3 model was trained on each of the 7-folds. Table-A1 shows that all of these 7 models provide good accuracy and weighted- score on their respective validation sets. Since these seven Inception-v3 models are not weak classifiers, one can use max-voting or average-voting on the outputs of these classifiers to make the final decision. To achieve this, we propose an intuitive ensembling that uses both max-voting and average-voting in a novel way. First, we consider the maximum confidence score on a test sample from the output of all seven classifiers. If this confidence score is more than a certain threshold , we consider it as the final decision and hence, use max-voting. If this confidence score is below

, we consider the harmonic mean of all the seven confidence scores as the final decision. We experimented with two values of

above 0.9 (, explained in detail in Sec-A.1). This is to note that the detection of true positives and true negatives is equally important in cancer diagnosis because a subject having cancer if left untreated may die. Likewise, a subject not having cancer, if stated to be suffering with cancer, may have traumatic and financial impact on the subject and family, besides health injury for giving unnecessary chemotherapy. Weighted- score is a reliable metric to evaluate the overall classifier’s performance, especially, when there is class imbalance in the dataset as it calculates score for each output class label and finds their average weighted by support (the number of true instances for each label). This alters ‘macro’ and ’micro’ score to account for label imbalance, which is also one of the challenges associated with this dataset.

6 Concluding Remarks

As shown in Table-3, we achieved a weighted score of 95.26% on the final test. (Refer to the submission on the leaderboard (ref50) with the username: shubham14100). Various recent research works discussed in Sec-1 used superior CNN architectures, say Resnet, ResNext, Densenet (isbi7) and ensemble of these architectures (isbi4) using cross-entropy loss to evaluate weighted score on CNMC-2019 dataset. However, our weighted score is so far highest on this dataset as reported in Table-3 because we have addressed the fundamental issue of heterogeneity in data by handling intersubject and intrasubject variability (subject-level heterogeneity) via incorporating it in sampling and the heterogeneity loss function (). Further, the ensemble approach gives better weighted score because this technique makes the overall inference more reliable as compared to the individual inference as shown in Table-A1. Our results indicate that the proposed method successfully handles intrasubject and intersubject heterogeneity and generalizes better on unseen data. Hence, the proposed method can be extended in a similar fashion to other applications that encounter some type of subject level heterogeneity in the dataset.

Figure 2: Left to Right: Long tail distribution of dataset, training, validation convergence plots for for random subjects sampled for training and validation set and T-SNE visualization of class-level on validation set of a randomly selected fold out seven folds. Convergence plots (with smoothness=0.8).
Comparison of different works Weighted-(%)
isbi7 84.90
isbi4 85.52
isbi3 86.6
isbi5 87.98
isbi8 91.04
gehlot2020sdct 94.86
Proposed 95.26
(ensemble with harmonic mean) 95.24
Table 3: Overall comparison on CNMC-2019 final test set.

Author Contributions

S.G., S.M. and D.S. contributed equally. A.G. and R.G. arranged funding and prepared the dataset. S.G., S.M., and D.S. proposed the method, coded and generated results. All authors discussed the results and prepared the manuscript.


Authors gratefully acknowledge the research funding support (Grant: EMR/2016/006183) from the Department of Science and Technology, Govt.of India for this research work. Authors would also like to acknowledge the support from Infosys Centre for AI, IIIT-Delhi, India.


Appendix A Appendix

Figure A1: TOP: Left to Right: Phase-I training with lr=, BOTTOM: Left to Right: Phase-II training with lr= . For both rows from Left to Right: training accuracy, validation accuracy, training loss and validation loss respectively, where loss is
on validation on test (with TTA)
model Accuracy (%) Weighted-(%) Weighted-(%)
Fold-0 model 95.25 95.26 92.72
Fold-1 model 94.60 94.59 94.41
Fold-2 model 92.97 92.97 93.63
Fold-3 model 83.78 89.58 92.13
Fold-4 model 94.96 94.96 91.43
Fold-5 model 86.81 86.82 89.82
Fold-6 model 94.492 94.93 90.39
Proposed 95.26
(ensemble with harmonic mean) 95.24
Table A1: Overall comparison of each fold’s model with the final proposed ensemble output

a.1 Choosing

The threshold is chosen as follows. First, we arbitrarily chose two high confidence values above 0.9 as 0.95 and 0.98. For each in Table-A2, we report the accuracy and weighted- on a subset of validation set corresponding to each fold in which the confidence output by their respective models is greater than . It is clear from Table-A2 that we can confidently use max-voting if the maximum confidence score for a test example is more than . We used harmonic mean for averaging the confidence scores if none of them is more than

as harmonic mean is more stable towards outliers.

Accuracy (%) Weighted- (%)
Fold-0 model 98.74 98.75
99.26 99.27
Fold-1 model 99.08 99.08
99.39 99.39
Fold-2 model 99.25 99.24
99.61 99.62
Fold-3 model 99.22 99.23
99.65 99.65
Fold-4 model 99.25 99.26
98.11 98.11
Fold-5 model 97.20 97.30
98.01 98.02
Fold-6 model 98.53 98.55
98.07 97.12
Table A2: Accuracy and Weighted- score on validation data having output confidence score more than , for each fold and