Deep ROC Analysis and AUC as Balanced Average Accuracy to Improve Model Selection, Understanding and Interpretation

03/21/2021
by   André M. Carrington, et al.
23

Optimal performance is critical for decision-making tasks from medicine to autonomous driving, however common performance measures may be too general or too specific. For binary classifiers, diagnostic tests or prognosis at a timepoint, measures such as the area under the receiver operating characteristic curve, or the area under the precision recall curve, are too general because they include unrealistic decision thresholds. On the other hand, measures such as accuracy, sensitivity or the F1 score are measures at a single threshold that reflect an individual single probability or predicted risk, rather than a range of individuals or risk. We propose a method in between, deep ROC analysis, that examines groups of probabilities or predicted risks for more insightful analysis. We translate esoteric measures into familiar terms: AUC and the normalized concordant partial AUC are balanced average accuracy (a new finding); the normalized partial AUC is average sensitivity; and the normalized horizontal partial AUC is average specificity. Along with post-test measures, we provide a method that can improve model selection in some cases and provide interpretation and assurance for patients in each risk group. We demonstrate deep ROC analysis in two case studies and provide a toolkit in Python.

READ FULL TEXT

page 13

page 14

06/08/2020

A Modified AUC for Training Convolutional Neural Networks: Taking Confidence into Account

Receiver operating characteristic (ROC) curve is an informative tool in ...
03/12/2019

ROC and AUC with a Binary Predictor: a Potentially Misleading Metric

In analysis of binary outcomes, the receiver operator characteristic (RO...
09/09/2020

Developing and Improving Risk Models using Machine-learning Based Algorithms

The objective of this study is to develop a good risk model for classify...
10/19/2020

Evaluating the incremental value of a new model: Area under the ROC curve or under the PR curve

Incremental value (IncV) evaluates the performance improvement from an e...
10/21/2021

Decision Theoretic Cutoff and ROC Analysis for Bayesian Optimal Group Testing

We study the inference problem in the group testing to identify defectiv...
06/14/2021

CathAI: Fully Automated Interpretation of Coronary Angiograms Using Neural Networks

Coronary heart disease (CHD) is the leading cause of adult death in the ...