ROC and AUC with a Binary Predictor: a Potentially Misleading Metric

03/12/2019
by   John Muschelli, et al.
0

In analysis of binary outcomes, the receiver operator characteristic (ROC) curve is heavily used to show the performance of a model or algorithm. The ROC curve is informative about the performance over a series of thresholds and can be summarized by the area under the curve (AUC), a single number. When a predictor is categorical, the ROC curve has only as many thresholds as the one less than number of categories; when the predictor is binary there is only one threshold. As the AUC may be used in decision-making processes on determining the best model, it important to discuss how it agrees with the intuition from the ROC curve. We discuss how the interpolation of the curve between thresholds with binary predictors can largely change the AUC. Overall, we believe a linear interpolation from the ROC curve with binary predictors, which is most commonly done in software, corresponding to the estimated AUC. We believe these ROC curves and AUC can lead to misleading results. We compare R, Python, Stata, and SAS software implementations.

READ FULL TEXT
research
06/08/2020

A Modified AUC for Training Convolutional Neural Networks: Taking Confidence into Account

Receiver operating characteristic (ROC) curve is an informative tool in ...
research
05/24/2022

Attributing AUC-ROC to Analyze Binary Classifier Performance

Area Under the Receiver Operating Characteristic Curve (AUC-ROC) is a po...
research
05/29/2023

The Misuse of AUC: What High Impact Risk Assessment Gets Wrong

When determining which machine learning model best performs some high im...
research
03/21/2021

Deep ROC Analysis and AUC as Balanced Average Accuracy to Improve Model Selection, Understanding and Interpretation

Optimal performance is critical for decision-making tasks from medicine ...
research
10/31/2019

Connecting population-level AUC and latent scale-invariant R^2 via Semiparametric Gaussian Copula and rank correlations

Area Under the Curve (AUC) is arguably the most popular measure of class...
research
07/29/2011

Technical Note: Towards ROC Curves in Cost Space

ROC curves and cost curves are two popular ways of visualising classifie...
research
11/29/2019

ROC movies – a new generalization to a popular classic

Throughout science and technology, receiver operating characteristic (RO...

Please sign up or login with your details

Forgot password? Click here to reset