Balanced-MixUp for Highly Imbalanced Medical Image Classification

Highly imbalanced datasets are ubiquitous in medical image classification problems. In such problems, it is often the case that rare classes associated to less prevalent diseases are severely under-represented in labeled databases, typically resulting in poor performance of machine learning algorithms due to overfitting in the learning process. In this paper, we propose a novel mechanism for sampling training data based on the popular MixUp regularization technique, which we refer to as Balanced-MixUp. In short, Balanced-MixUp simultaneously performs regular (i.e., instance-based) and balanced (i.e., class-based) sampling of the training data. The resulting two sets of samples are then mixed-up to create a more balanced training distribution from which a neural network can effectively learn without incurring in heavily under-fitting the minority classes. We experiment with a highly imbalanced dataset of retinal images (55K samples, 5 classes) and a long-tail dataset of gastro-intestinal video frames (10K images, 23 classes), using two CNNs of varying representation capabilities. Experimental results demonstrate that applying Balanced-MixUp outperforms other conventional sampling schemes and loss functions specifically designed to deal with imbalanced data. Code is released at https://github.com/agaldran/balanced_mixup .

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 2

04/15/2021

Out-of-Distribution Detection for Dermoscopic Image Classification

Medical image diagnosis can be achieved by deep neural networks, provide...
06/17/2021

MetaBalance: High-Performance Neural Networks for Class-Imbalanced Data

Class-imbalanced data, in which some classes contain far more samples th...
11/25/2019

A Self-Adaptive Synthetic Over-Sampling Technique for Imbalanced Classification

Traditionally, in supervised machine learning, (a significant) part of t...
04/07/2020

Long-Tailed Recognition Using Class-Balanced Experts

Classic deep learning methods achieve impressive results in image recogn...
11/12/2020

Improving Model Accuracy for Imbalanced Image Classification Tasks by Adding a Final Batch Normalization Layer: An Empirical Study

Some real-world domains, such as Agriculture and Healthcare, comprise ea...
11/02/2017

Oversampling for Imbalanced Learning Based on K-Means and SMOTE

Learning from class-imbalanced data continues to be a common and challen...
11/03/2020

Image Classification via Quantum Machine Learning

Quantum Computing and especially Quantum Machine Learning, in a short pe...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Backed by the emergence of increasingly powerful convolutional neural networks, medical image classification has made remarkable advances over the last years, reaching unprecedented levels of accuracy

[20]. However, due to the inherent difficulty in collecting labeled examples of rare diseases or other unusual instances in a medical context, these models are often trained and tested on large datasets containing a more balanced distribution with respect to image classes than the one found in real-life clinical scenarios, which typically has a long-tailed distribution. Under such severe data imbalance, over-represented (i.e., majority) classes tend to dominate the training process, resulting in a decrease of performance in under-represented (i.e., minority) classes [31, 23]. Therefore, developing training methods that are adapted to strong data imbalance is essential for the advance of medical image classification.

Common solutions to address data imbalance involve data re-sampling to achieve a balanced class distribution [4], curriculum learning [13], adapted loss functions, e.g. cost-sensitive classification [30, 9], or weighting the contribution of the different samples [7, 19]. Another approach uses synthetic manipulation of data and/or labels to drive the learning process towards a more suitable solution, like label smoothing [8] or SMOTE [5]. Our proposed technique is a combination of modified training data sampling strategies with synthetic data manipulation via the well-known MixUp regularization method [28]. Therefore it is deeply connected with MixUp, with the fundamental difference that, while MixUp randomly combines training samples without taking into account their classes, we carefully mix examples from minority categories with examples from the other classes in order to create a more suitable training data distribution.

(a)
(b)
Figure 3: The two different data imbalance scenarios considered in this paper. Left: Highly imbalanced problem (DR grading from retinal images, [17]). Right: Long-tailed data (Gastro-intestinal image classification, [2]).

Some approaches have been recently explored involving MixUp and data imbalance. ReMix [6] mixes up by keeping the minority class label, instead of mixing up the labels. Similarly, MixBoost [14]

attempts to combine active learning with MixUp to select which training samples to mix from each category, adding an extra complexity layer to the sampling process. Another popular technique that is related to our approach is SMOTE

[5]. However, SMOTE generates convex combinations of input samples only between nearest neighbors of the same class. While this creates extra examples in regions of the space where there is less data, it does not provide the same regularizing benefits as MixUp.

In this paper, we propose Balanced-MixUp, a new imbalanced-robust training method that mixes up imbalanced (instance-based) and balanced (class-based) sampling of the data. Experiments on two different medical image classification tasks with highly imbalanced and long-tailed data (as shown in Fig. 3), using neural networks of different complexities, show that Balanced-MixUp is effective in creating a more evenly distributed training data and also regularizing overparametrized models, resulting in better performance than other conventional and recent approaches to handle data imbalance in all the considered cases.

2 Methodology

In this paper we intend to combine the MixUp regularization technique with a modified sampling strategy for imbalanced datasets. This section introduces each of these concepts and then our proposed combined technique.

2.1 MixUp Regularization

The MixUp technique was initially introduced in [28] as a simple regularization technique to decrease overfitting in deep neural networks. If we denote a training example composed of an image

and its associated one-hot encoded label

, with being a neural network to be trained to approximate the mapping , then MixUp creates synthetic examples and labels as follows:

(1)

where , with . Since , and are random convex combinations of data and label inputs. Typical values of vary in , which means that in practice will likely be relatively close to either or . Despite its simplicity, it has been shown that optimizing on mixed-up data leads to better generalization and improves model calibration [26].

2.2 Training Data Sampling Strategies

When dealing with extremely imbalanced data, the training process is typically impacted by different per-class learning dynamics that result in underfitting of minority classes, which are rarely presented to the model and may end up being entirely ignored [15]. Modified sampling strategies can be utilized to mitigate this effect, like oversampling under-represented categories, although merely doing so often leads to counter-productive outcomes, e.g. repeatedly showing to the model the same training examples may lead to the overfitting of minority classes [5].

Before detailing how to combine oversampling of minority classes with MixUp regularization, we introduce some further notation to describe sampling strategies. Given a training set for a multi-class problem with classes, if each class contains examples, we have that . We can then describe common data sampling strategies as follows:

(2)

being

the probability of sampling from class

during training. With this, selecting amounts to picking examples with a probability equal to the frequency of their class in the training set (instance-based sampling), whereas choosing leads to a uniform probability of sampling from each class, this is, class-based sampling or oversampling of minority classes. Another popular choice is square-root sampling, which stems from selecting .

Figure 4: Schematic illustration of Balanced-MixUp (right) filling underpopulated areas in the data space - linked to infrequent categories - with synthetic examples.

2.3 Balanced-MixUp

We propose to combine the two training enhancements described above into a single technique that is suitable for learning under strong data imbalance. We introduce an adaptation to this scenario of MixUp, referred to as Balanced-MixUp, in which two data points are not randomly sampled without considering their category, but rather sampling one following an instance-based strategy and the other following a class-based sampling.

Let us refer to a training dataset together with a sampling strategy as , with and referring to instance-based and class-based sampling respectively. With this notation, Balanced-MixUp can be described as follows:

(3)

for . This induces a more balanced distribution of training examples by creating synthetic data points around regions of the space where minority classes provide less data density, as illustrated in Fig. 4. At the same time, adding noise to the labels helps regularizing the learning process.

In our case, departing from the original MixUp formulation, we obtain the mixing coefficient as . This results in an exponential-like distri-

Figure 5: distribution for varying

bution as shown in Fig. 5, which leads to convex combinations in which examples from

receive more weight, preventing overfitting in minority classes. In addition, it allows us to formulate our technique as depending on a single hyperparameter with an intuitive behavior: as

grows, examples from minority classes are mixed up with greater weights, increasing class balance at the expense of risking some overfitting.

2.4 Training Details

All models in this paper are CNNs implemented in Pytorch

[22]

and optimized to reduce the Cross-Entropy loss using Stochastic Gradient Descent with a batch size of

and a learning rate of . During training, standard data augmentation operations are applied, and the learning rate is cyclically annealed towards 0. The metric of interest in each task (see next section) is used to monitor model’s performance on the validation set and the best one is kept. After training, we generate predictions by averaging the result of the input image and its horizontally flipped version, which is common practice to regularize predictions [25].

3 Experimental Analysis

In this section we detail our experimental setup: considered datasets with relevant statistics, baseline techniques for comparison, evaluation metrics, and a discussion on the numerical differences in performance between methods.

3.1 Experimental Details

We consider two different image classification tasks: Diabetic Retinopathy (DR) grading from retinal fundus images and gastrointestinal image (GI) classification.

  • [leftmargin=*]

  • For DR grading, we use the Eyepacs database111https://www.kaggle.com/c/diabetic-retinopathy-detection the largest publicly available dataset for this task. This database has images labeled as belonging to one out of five possible DR grades, ranging from DR0 (no DR) to DR4 (proliferative DR). Due to the nature of this disease, and the very subtle signs associated to the DR1 class (just a few microaneurysms present), the class distribution is highly imbalanced, as shown in Fig. 3. The database contains 35,000 training examples and 55,000 images assigned to a separate test set. We use of the training set for validation and early-stopping purposes, and derive performance measures from the official test set. As an external database to examine model generalization we also use the 1748 retinal images in the Messidor-2 dataset222https://www.adcis.net/en/third-party/messidor2/, with the grades provided333https://www.kaggle.com/google-brain/messidor2-dr-grades in [17].

  • For GI image classification, we use the Hyper-Kvasir dataset444https://endotect.com/, recently made available [2]. This database contains 10,662 images labeled with anatomical landmarks as well as pathological and normal findings, and it represents a challenging classification problem due to the larger amount of classes (23) and the varying class frequencies, with minority classes very rarely represented, see Fig. 3. Please note that there is also an official test set [12], but we found this to contain only 11 different classes. Since our focus is on assessing model performance under a long-tail distribution, we prefer to discard this set, running all our experiments and computing performance for all models in a stratified 5-fold cross-validation manner.

For performance assessment in the DR case, we adopt as the main evaluation a measure based on the quadratic-weighted kappa score (quad-), which is the standard performance measure in grading problems, as it considers the correct ordering of predictions. We also report Matthews Correlation Coefficient (MCC), which has been shown to be meaningful under strong class imbalance [3], and the Kendall- coefficient. For the GI image classification task, we adopt as the main assessment measure MCC, as it was used for performance ranking in a recent challenge associated to this database [12]

. Since we are dealing with a long-tail data distribution, we also report Balanced Accuracy (the mean of the diagonal in a normalized confusion matrix) and the macro-F1 score, which capture well performance in minority classes.

For comparison purposes, in both tasks we contrast Balanced-MixUp with other conventional sampling approaches: 1) Class-based sampling (oversampling of minority classes, as recommended in [4]), 2) Instance-based sampling, and 3) sampling based on the square root of class-frequencies [21]. We also include training with loss functions that are meant to perform better under severe class imbalance, namely 4) the popular Focal Loss [18], which modulates the contribution of each example according to the amount of loss it generates, and 5) a Class-Balanced loss recently introduced that weighs the loss of each class during training based on the effective number of samples, see [7].

In all cases, in order to observe the behavior of the different techniques with models of varying sizes, we train two different CNN baseline architectures: a MobileNet V2 [24] and more powerful ResNext50, modified as in [16]. For sensitivity analysis purposes, we measure the impact of varying the only hyperparameter in Balanced-MixUp, with and report the corresponding results.

3.2 Discussion on the Numerical Results on DR grading

Assessment measures for the task of DR grading from retinal images are displayed in Table 1. The leftmost three columns show performance of a MobileNet V2 CNN in terms of quad-, MCC and Kendall-. In this case, it can be seen that Balanced-MixUp with outperforms all other approaches in terms of the three considered figures of merit. Note that setting still outperforms all other techniques. It is also interesting to observe the differences when we switch to a larger model, ResNext50. In particular, we can see that: 1) the performance of almost all methods is increased, indicating that ResNext50 is a more suitable choice for this problem; and 2) Balanced-MixUp with is now the dominant approach, reaching the highest scores in all considered measures. This is consistent with the intuition that Balanced-MixUp can regularize the learning process, and that increasing its hyperparameter leads to a stronger regularization Indeed, for this architecture the second best method is Balanced-MixUp with . It is also worth stressing that, again, the three values of translate in better results than any of the other considered alternatives.

MobileNet V2 ResNeXt-50
quad- MCC Kendall- quad- MCC Kendall-
Class-sampling 74.58 49.34 63.30 74.45 51.12 64.14
Instance-sampling 78.75 61.79 73.39 80.17 62.78 74.32
Sqrt-sampling [21] 79.32 59.66 72.22 79.38 58.77 72.22
Focal Loss [18] 78.59 60.74 72.70 79.73 62.66 73.37
CB Loss [7] 77.84 61.08 72.85 79.19 61.83 74.32
Bal-Mxp: 79.61 62.41 74.05 80.35 63.31 74.74
Bal-Mxp: 79.43 62.22 73.69 80.78 63.78 75.05
Bal-Mxp: 78.34 61.65 73.25 80.62 63.51 74.89
Table 1: Performance comparison with two different CNN architectures for DR grading on the Eyepacs test set. Best results are marked bold.

It is important to stress that the results shown in Table 1 do not represent the state-of-the-art in this dataset. Recent methods are capable of reaching greater quad- values by means of specialized architectures with attention mechanisms [11]

, or semi-supervised learning with segmentation branches

[29]. Our results are however aligned with other recently published techniques, as shown in Table 2, which also contains test results in the Messidor-2 dataset (without any re-training), demonstrating their competitiveness and generalization ability. Moreover we consider Balanced-MixUp to be a supplementary regularizing technique of the learning process from which other methods could also benefit.

QWKL
PRL’18 [27]
MedIA’20 [1]
Cost-Sensitive
MICCAI’20 [9]
Iter. Aug.
TMI’20 [10]
Bal-Mxp
Rx50,
Eyepacs 74.00 74.00 78.71 80.00 80.78
Messidor-2 —– 71.00 79.79 —– 85.14
Table 2: Performance comparison in terms of quad- for methods trained on the Eyepacs dataset and tested (no retraining) on Eyepacs and Messidor-2.

3.3 Discussion on the Numerical Results on GI image classification

The second task we consider is Gastro-Intestinal image classification. Here the class imbalance is greater than in the DR problem, in addition to the presence of 23 categories, which turn this into a more challenging dataset. Table 3 shows the median performance in a stratified 5-fold cross-validation process, when considering exactly the same techniques and architectures as in the previous section.

Although in this task the performance measures appear to be more mixed than in the DR problem, we can still see how Balanced-MixUp appears to bring improvements over the other competing approaches. Again, the ResNext50 architecture delivers better performance in terms of MCC, although differences are not as high as before, and some methods show a degraded performance when applied with this larger architecture (compared to using MobileNet).

Focusing first on the MCC score, we can appreciate again the regularizing effect of Balanced-MixUp: for both architectures, applying it leads to improved performance. Moreover, we can again see the same trend as above: the MobileNet model performs better with less regularization (), and increasing slowly deteriorates the MCC. As expected, when using the ResNeXt architecture, a greater delivers better performance. It is worth noting that the MCC reached by the MobileNet model with is greater than in any of the considered methods and architectures, excluding models regularized with Balanced-MixUp and varying .

Finally, it is relevant to observe that when we consider balanced accuracy and macro-F1, which can capture better performance in minority classes, for almost all methods it appears to be beneficial to use a smaller architecture. This appears to reduce to some extent the minority class overfitting due to oversampling, observed in the low performance of class-based sampling. Interestingly, even with this small architecture, Balanced MixUp increases the performance for all values of , which further shows that our approach not only helps regularizing large models but also contributes to an improved learning on minority classes.

MobileNet V2 ResNeXt-50
MCC B-ACC Macro-F1 MCC B-ACC Macro-F1
Class-sampling 89.74 61.84 61.84 88.97 59.57 58.40
Instance-sampling 90.49 62.13 62.16 90.74 61.41 61.69
Sqrt-sampling [21] 90.11 63.01 62.66 90.29 62.84 62.85
Focal Loss [18] 90.30 62.09 61.78 90.23 62.12 62.36
CB Loss [7] 85.84 54.93 54.87 89.67 63.84 63.71
Bal-Mxp: 90.90 63.49 62.77 91.05 62.55 62.92
Bal-Mxp: 90.54 63.44 63.85 91.15 62.80 64.00
Bal-Mxp: 90.39 64.76 64.07 90.84 62.34 62.35
Table 3: Stratified 5-Fold cross-validation (median) results for GI image classification on the Hyper-Kvasir dataset. Best results are marked bold.

4 Conclusion

Imbalanced data is present in almost any medical image analysis task. Therefore, designing techniques for learning in such regimes is a challenge of great clinical significance. This paper introduces Balanced-MixUp to deal with heavy imbalanced data distributions by means of the combination of the popular MixUp regularization technique and modified training data sampling strategies. Balanced-MixUp is easy to implement and delivers consistent performance improvements when compared with other popular techniques. Its extension to other tasks beyond image classification represents a promising future research direction.

Acknowledgments

This work was partially supported by a Marie Skłodowska-Curie Global Fellowship (No 892297) and by Australian Research Council grants (DP180103232 and FT190100525).

References

  • [1]

    Araujo, T., Aresta, G., Mendonça, L., Penas, S., Maia, C., Carneiro, A., Mendonça, A.M., Campilho, A.: DR|GRADUATE: Uncertainty-aware deep learning-based diabetic retinopathy grading in eye fundus images. Medical Image Analysis

    63, 101715 (Jul 2020)
  • [2] Borgli, H., Thambawita, V., Smedsrud, P.H., Hicks, S., Jha, D., Eskeland, S.L., Randel, K.R., Pogorelov, K., Lux, M., Nguyen, D.T.D., Johansen, D., Griwodz, C., Stensland, H.K., Garcia-Ceja, E., Schmidt, P.T., Hammer, H.L., Riegler, M.A., Halvorsen, P., de Lange, T.: HyperKvasir , a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Scientific Data 7(1),  283 (Aug 2020)
  • [3]

    Boughorbel, S., Jarray, F., El-Anbari, M.: Optimal classifier for imbalanced data using Matthews Correlation Coefficient metric. PLOS ONE

    12(6) (Jun 2017)
  • [4] Buda, M., Maki, A., Mazurowski, M.A.: A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks 106, 249–259 (Oct 2018)
  • [5]

    Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: SMOTE: synthetic minority over-sampling technique. Journal of Artificial Intelligence Research

    16(1), 321–357 (Jun 2002)
  • [6]

    Chou, H.P., Chang, S.C., Pan, J.Y., Wei, W., Juan, D.C.: Remix: Rebalanced Mixup. In: Computer Vision – ECCV 2020 Workshops. pp. 95–110. Cham (2020)

  • [7]

    Cui, Y., Jia, M., Lin, T.Y., Song, Y., Belongie, S.: Class-Balanced Loss Based on Effective Number of Samples. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (Jun 2019)

  • [8] Galdran, A., Chelbi, J., Kobi, R., Dolz, J., Lombaert, H., Ayed, I.b., Chakor, H.: Non-uniform Label Smoothing for Diabetic Retinopathy Grading from Retinal Fundus Images with Deep Neural Networks. Translational Vision Science & Technology 9(2), 34–34 (Jan 2020)
  • [9] Galdran, A., Dolz, J., Chakor, H., Lombaert, H., Ben Ayed, I.: Cost-Sensitive Regularization for Diabetic Retinopathy Grading from Eye Fundus Images. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. pp. 665–674. Lecture Notes in Computer Science (2020)
  • [10] González-Gonzalo, C., Liefers, B., Ginneken, B.v., Sánchez, C.I.: Iterative Augmentation of Visual Evidence for Weakly-Supervised Lesion Localization in Deep Interpretability Frameworks: Application to Color Fundus Images. IEEE Transactions on Medical Imaging 39(11), 3499–3511 (Nov 2020)
  • [11] He, A., Li, T., Li, N., Wang, K., Fu, H.: CABNet: Category Attention Block for Imbalanced Diabetic Retinopathy Grading. IEEE Transactions on Medical Imaging 40(1), 143–153 (Jan 2021)
  • [12] Hicks, S., Jha, D., Thambawita, V., Halvorsen, P., Hammer, H.L., Riegler, M.: The EndoTect 2020 Challenge: Evaluation and Comparison of Classification, Segmentation and Inference Time for Endoscopy. In: 25th International Conference on Pattern Recognition (ICPR) (2020)
  • [13] Jiménez-Sánchez, A., Mateus, D., Kirchhoff, S., Kirchhoff, C., Biberthaler, P., Navab, N., González Ballester, M.A., Piella, G.: Medical-based Deep Curriculum Learning for Improved Fracture Classification. In: MICCAI 2019. pp. 694–702. Lecture Notes in Computer Science (2019)
  • [14] Kabra, A., Chopra, A., Puri, N., Badjatiya, P., Verma, S., Gupta, P., K, B.: MixBoost: Synthetic Oversampling with Boosted Mixup for Handling Extreme Imbalance (Sep 2020), http://arxiv.org/abs/2009.01571, arXiv: 2009.01571
  • [15] Kang, B., Xie, S., Rohrbach, M., Yan, Z., Gordo, A., Feng, J., Kalantidis, Y.: Decoupling representation and classifier for long-tailed recognition. In: ICLR (2020)
  • [16] Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., Houlsby, N.: Big Transfer (BiT): General Visual Representation Learning. In: Computer Vision – ECCV 2020. pp. 491–507 (2020)
  • [17] Krause, J., Gulshan, V., Rahimy, E., Karth, P., Widner, K., Corrado, G.S., Peng, L., Webster, D.R.: Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy. Ophthalmology 125(8), 1264–1272 (Aug 2018)
  • [18] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal Loss for Dense Object Detection. pp. 2980–2988 (2017)
  • [19] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal Loss for Dense Object Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(2), 318–327 (Feb 2020)
  • [20] Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., van der Laak, J.A.W.M., van Ginneken, B., Sánchez, C.I.: A survey on deep learning in medical image analysis. Medical Image Analysis 42, 60–88 (Dec 2017)
  • [21] Mahajan, D.K., Girshick, R.B., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., Maaten, L.v.d.: Exploring the Limits of Weakly Supervised Pretraining. In: ECCV (2018)
  • [22] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: PyTorch: An Imperative Style, High-Performance Deep Learning Library. In: NEURIPS 2019, pp. 8024–8035
  • [23] Quellec, G., Lamard, M., Conze, P.H., Massin, P., Cochener, B.: Automatic detection of rare pathologies in fundus photographs using few-shot learning. Medical Image Analysis 61, 101660 (Apr 2020)
  • [24] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: MobileNetV2: Inverted Residuals and Linear Bottlenecks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4510–4520 (Jun 2018)
  • [25] Shanmugam, D., Blalock, D., Balakrishnan, G., Guttag, J.: When and Why Test-Time Augmentation Works (Nov 2020), http://arxiv.org/abs/2011.11156, arXiv: 2011.11156
  • [26] Thulasidasan, S., Chennupati, G., Bilmes, J.A., Bhattacharya, T., Michalak, S.: On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks. In: Advances in Neural Information Processing Systems (2019)
  • [27] de la Torre, J., Puig, D., Valls, A.: Weighted kappa loss function for multi-class classification of ordinal data in deep learning. Pattern Recognition Letters 105, 144–154 (Apr 2018)
  • [28] Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond Empirical Risk Minimization. In: International Conference on Learning Representations (2018)
  • [29] Zhou, Y., He, X., Huang, L., Liu, L., Zhu, F., Cui, S., Shao, L.: Collaborative Learning of Semi-Supervised Segmentation and Classification for Medical Images. In: Conference on Computer Vision and Pattern Recognition (Jun 2019)
  • [30] Zhou, Z.H., Liu, X.Y.: Training cost-sensitive neural networks with methods addressing the class imbalance problem. IEEE Transactions on Knowledge and Data Engineering 18(1), 63–77 (Jan 2006)
  • [31] Zhuang, J., Cai, J., Wang, R., Zhang, J., Zheng, W.: CARE: Class Attention to Regions of Lesion for Classification on Imbalanced Data. In: International Conference on Medical Imaging with Deep Learning. pp. 588–597. PMLR (May 2019)

Appendix 0.A Appendix: Extended Numerical Results

0.a.1 Numerical Results on DR grading

For deriving confidence intervals, labels and model predictions on the Eyepacs test set are bootstrapped (n=1000) in a stratified manner according to the presence of DR grades. Tables

4 and 5 show the mean/std resulting from this.

MobileNet V2
quad- MCC Kendall-
Class-sampling 74.57 49.33 63.29
Instance-sampling 78.74 61.79 73.38
Sqrt-sampling 79.32 59.66 72.21
Focal Loss 78.58 60.74 72.68
CB Loss 77.83 61.07 72.84
Bal-Mxp:
Bal-Mxp: 79.42 62.23 73.68
Bal-Mxp: 78.34 61.65 73.24
Table 4: Performance comparison on Eyepacs. Best results marked bold.
ResNeXt-50
quad- MCC Kendall-
Class-sampling 74.45 51.13 64.14
Instance-sampling 80.17 62.78 74.31
Sqrt-sampling 79.39 58.78 72.22
Focal Loss 79.73 62.67 74.32
CB Loss 79.18 61.84 73.36
Bal-Mxp: 80.35 63.30 74.74
Bal-Mxp:
Bal-Mxp: 80.62 63.50 74.89
Table 5: Performance comparison on Eyepacs. Best results marked bold.

0.a.2 Numerical Results on GI image classification

Results on GI image classification in our paper were obtained by a stratified 5-fold cross-validation process, due to the absence of data from minority classes in the official test set. In the paper we reported the median performance across folds, but it is also useful to observe performance dispersion. Tables 6 and 7 show results in terms of the minimum, median, and maximum performance across folds for each model, displayed as minimum median maximum values.

MobileNet V2
MCC B-ACC Macro-F1
Class-sampling 89.2289.7490.23 61.6389.8463.13 61.1461.8466.42
Instance-sampling 90.3490.4991.47 60.9262.1364.73 60.0262.1664.23
Sqrt-sampling 89.8890.1190.96 61.1063.0165.49 61.3762.6671.28
Focal Loss 90.1490.3090.60 61.5562.0964.70 61.0361.7864.45
CB Loss 84.0985.8486.33 53.5054.9356.74 53.4354.8756.01
Bal-Mxp 90.1490.9091.04 61.5763.4969.60 61.5462.7766.48
Bal-Mxp 90.0190.5490.91 60.8663.4469.85 60.4063.8568.50
Bal-Mxp 89.8590.3990.89 62.1064.7669.89 61.5564.0769.48
Table 6: Stratified 5-Fold cross-validation (min, median, max) results for GI image classification on the Hyper-Kvasir dataset. Best results are marked bold.
ResNeXt-50
MCC B-ACC Macro-F1
Class-sampling 88.9789.8490.28 59.5761.3864.05 58.4061.5264.14
Instance-sampling 90.6090.7491.10 60.0561.4162.08 59.7261.6962.32
Sqrt-sampling 89.7890.2990.49 60.7062.8466.71 60.3962.8566.21
Focal Loss 89.9390.2391.51 60.2362.1264.08 60.2962.3664.78
CB Loss 89.6389.6790.29 61.9963.8468.81 62.2263.7168.66
Bal-Mxp 90.3891.0591.20 60.4462.5564.37 60.9062.9264.52
Bal-Mxp 90.5991.1591.20 60.7062.8065.32 61.3564.0065.86
Bal-Mxp 90.4990.8491.30 60.1462.3465.89 60.3462.3565.62
Table 7: Stratified 5-Fold cross-validation (min, median, max) results for GI image classification on the Hyper-Kvasir dataset. Best results are marked bold.

Due to the few instances in the minority classes, results for the GI classification task are noisier than for the DR grading problem. However, one can see (as pointed out in the paper) that for metrics that highlight performance on the minority classes, like Balanced Accuracy or Macro-F1, not only are the overall best median results obtained by the MobileNet model with Balanced-MixUp regularization (with ), but also the best minimum and maximum values are attained by this combination, although it is worth noting the good performance of the CB-loss in conjunction with the ResNeXt50 architecture.