Improving Trustworthiness of AI Disease Severity Rating in Medical Imaging with Ordinal Conformal Prediction Sets

07/05/2022
by   Charles Lu, et al.
0

The regulatory approval and broad clinical deployment of medical AI have been hampered by the perception that deep learning models fail in unpredictable and possibly catastrophic ways. A lack of statistically rigorous uncertainty quantification is a significant factor undermining trust in AI results. Recent developments in distribution-free uncertainty quantification present practical solutions for these issues by providing reliability guarantees for black-box models on arbitrary data distributions as formally valid finite-sample prediction intervals. Our work applies these new uncertainty quantification methods – specifically conformal prediction – to a deep-learning model for grading the severity of spinal stenosis in lumbar spine MRI. We demonstrate a technique for forming ordinal prediction sets that are guaranteed to contain the correct stenosis severity within a user-defined probability (confidence interval). On a dataset of 409 MRI exams processed by the deep-learning model, the conformal method provides tight coverage with small prediction set sizes. Furthermore, we explore the potential clinical applicability of flagging cases with high uncertainty predictions (large prediction sets) by quantifying an increase in the prevalence of significant imaging abnormalities (e.g. motion artifacts, metallic artifacts, and tumors) that could degrade confidence in predictive performance when compared to a random sample of cases.

READ FULL TEXT

page 6

page 9

research
07/15/2021

A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification

Black-box machine learning learning methods are now routinely used in hi...
research
06/01/2023

Quantifying Deep Learning Model Uncertainty in Conformal Prediction

Precise estimation of predictive uncertainty in deep neural networks is ...
research
09/09/2021

Fair Conformal Predictors for Applications in Medical Imaging

Deep learning has the potential to augment many components of the clinic...
research
06/01/2021

Locally Valid and Discriminative Confidence Intervals for Deep Learning Models

Crucial for building trust in deep learning models for critical real-wor...
research
09/29/2020

Uncertainty Sets for Image Classifiers using Conformal Prediction

Convolutional image classifiers can achieve high predictive accuracy, bu...
research
06/15/2022

Leveraging Uncertainty in Deep Learning for Pancreatic Adenocarcinoma Grading

Pancreatic cancers have one of the worst prognoses compared to other can...
research
04/27/2020

Calibrating Healthcare AI: Towards Reliable and Interpretable Deep Predictive Models

The wide-spread adoption of representation learning technologies in clin...

Please sign up or login with your details

Forgot password? Click here to reset