A Comparative Study of Confidence Calibration in Deep Learning: From Computer Vision to Medical Imaging

06/17/2022
by   Riqiang Gao, et al.
0

Although deep learning prediction models have been successful in the discrimination of different classes, they can often suffer from poor calibration across challenging domains including healthcare. Moreover, the long-tail distribution poses great challenges in deep learning classification problems including clinical disease prediction. There are approaches proposed recently to calibrate deep prediction in computer vision, but there are no studies found to demonstrate how the representative models work in different challenging contexts. In this paper, we bridge the confidence calibration from computer vision to medical imaging with a comparative study of four high-impact calibration models. Our studies are conducted in different contexts (natural image classification and lung cancer risk estimation) including in balanced vs. imbalanced training sets and in computer vision vs. medical imaging. Our results support key findings: (1) We achieve new conclusions which are not studied under different learning contexts, e.g., combining two calibration models that both mitigate the overconfident prediction can lead to under-confident prediction, and simpler calibration models from the computer vision domain tend to be more generalizable to medical imaging. (2) We highlight the gap between general computer vision tasks and medical imaging prediction, e.g., calibration methods ideal for general computer vision tasks may in fact damage the calibration of medical imaging prediction. (3) We also reinforce previous conclusions in natural image classification settings. We believe that this study has merits to guide readers to choose calibration models and understand gaps between general computer vision and medical imaging domains.

READ FULL TEXT

page 2

page 7

page 8

page 9

page 10

page 13

07/06/2021

Confidence-based Out-of-Distribution Detection: A Comparative Study and Analysis

Image classification models deployed in the real world may receive input...
06/02/2022

Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives

Transformer, the latest technological advance of deep learning, has gain...
05/27/2022

Failure Detection in Medical Image Classification: A Reality Check and Benchmarking Testbed

Failure detection in automated image classification is a critical safegu...
09/27/2017

A Comparative Study of CNN, BoVW and LBP for Classification of Histopathological Images

Despite the progress made in the field of medical imaging, it remains a ...
09/09/2020

Improved Trainable Calibration Method for Neural Networks on Medical Imaging Classification

Recent works have shown that deep neural networks can achieve super-huma...
09/22/2020

Improving Medical Annotation Quality to Decrease Labeling Burden Using Stratified Noisy Cross-Validation

As machine learning has become increasingly applied to medical imaging d...
08/05/2021

Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications

Recently, there have been breakthroughs in computer vision ("CV") models...