Model-based ROC (mROC) curve: examining the effect of case-mix and model calibration on the ROC plot

by   Mohsen Sadatsafavi, et al.

The performance of a risk prediction model is often characterized in terms of discrimination and calibration. The Receiver Operating Characteristic (ROC) curve is widely used for evaluating model discrimination. When comparing the ROC curves between the development and an independent (external) validation sample, the effect of case-mix makes the interpretation of discrepancies difficult. Further, compared to discrimination, evaluating calibration has not received the same level of attention in the medical literature. The most commonly used graphical method for model calibration, the calibration plot, involves smoothing or grouping of the data, requiring arbitrary specification of smoothing parameters or the number of groups. In this work, we introduce the 'model-based' ROC (mROC) curve, the ROC curve that should be observed if the prediction model is calibrated in the external population. We first show that moderate calibration (having a response probability of p condition for convergence of the empirical ROC and mROC curves. We further show that equivalence of the expected values of the predicted and observed risk (mean calibration, or calibration-in-the-large) and equivalence of the mROC and ROC curves together guarantee moderate calibration. We demonstrate how mROC separates the effect of case-mix and model mis-calibration when comparing ROC curves from different samples. We also propose a test statistic for moderate calibration, which does not require any arbitrary parameterization. We conduct simulations to assess small-sample properties of the proposed test. A case study puts these developments in a practical context. We conclude that mROC can easily be constructed and used to interpret the effect of case-mix on the ROC curve and to evaluate model calibration on the ROC plot.



There are no comments yet.


page 17

page 34


Real-time imputation of missing predictor values in clinical practice

Use of prediction models is widely recommended by clinical guidelines, b...

Honest calibration assessment for binary outcome predictions

Probability predictions from binary regressions or machine learning meth...

Non-isometric Curve to Surface Matching with Incomplete Data for Functional Calibration

Calibration refers to the process of adjusting features of a computation...

Inference for ROC Curves Based on Estimated Predictive Indices

We provide a comprehensive theory of conducting in-sample statistical in...

Accommodating heterogeneous missing data patterns for prostate cancer risk prediction

Objective: We compared six commonly used logistic regression methods for...

Receiver Operating Characteristic (ROC) Curves

Receiver operating characteristic (ROC) curves are used ubiquitously to ...

Minding non-collapsibility of odds ratios when recalibrating risk prediction models

In clinical prediction modeling, model updating refers to the practice o...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.