Model-based ROC (mROC) curve: examining the effect of case-mix and model calibration on the ROC plot

02/29/2020
by   Mohsen Sadatsafavi, et al.
0

The performance of a risk prediction model is often characterized in terms of discrimination and calibration. The Receiver Operating Characteristic (ROC) curve is widely used for evaluating model discrimination. When comparing the ROC curves between the development and an independent (external) validation sample, the effect of case-mix makes the interpretation of discrepancies difficult. Further, compared to discrimination, evaluating calibration has not received the same level of attention in the medical literature. The most commonly used graphical method for model calibration, the calibration plot, involves smoothing or grouping of the data, requiring arbitrary specification of smoothing parameters or the number of groups. In this work, we introduce the 'model-based' ROC (mROC) curve, the ROC curve that should be observed if the prediction model is calibrated in the external population. We first show that moderate calibration (having a response probability of p condition for convergence of the empirical ROC and mROC curves. We further show that equivalence of the expected values of the predicted and observed risk (mean calibration, or calibration-in-the-large) and equivalence of the mROC and ROC curves together guarantee moderate calibration. We demonstrate how mROC separates the effect of case-mix and model mis-calibration when comparing ROC curves from different samples. We also propose a test statistic for moderate calibration, which does not require any arbitrary parameterization. We conduct simulations to assess small-sample properties of the proposed test. A case study puts these developments in a practical context. We conclude that mROC can easily be constructed and used to interpret the effect of case-mix on the ROC curve and to evaluate model calibration on the ROC plot.

READ FULL TEXT

page 17

page 34

research
07/19/2023

Non-parametric inference on calibration of predicted risks

Moderate calibration, the expected event probability among observations ...
research
01/25/2023

Evaluating Probabilistic Classifiers: The Triptych

Probability forecasts for binary outcomes, often referred to as probabil...
research
08/05/2015

Non-isometric Curve to Surface Matching with Incomplete Data for Functional Calibration

Calibration refers to the process of adjusting features of a computation...
research
07/28/2023

Is this model reliable for everyone? Testing for strong calibration

In a well-calibrated risk prediction model, the average predicted probab...
research
09/13/2022

Evaluating individualized treatment effect predictions: a new perspective on discrimination and calibration assessment

Personalized medicine constitutes a growing area of research that benefi...
research
08/25/2023

Calibration plots for multistate risk predictions models: an overview and simulation comparing novel approaches

Introduction. There is currently no guidance on how to assess the calibr...
research
09/21/2021

Accommodating heterogeneous missing data patterns for prostate cancer risk prediction

Objective: We compared six commonly used logistic regression methods for...

Please sign up or login with your details

Forgot password? Click here to reset