Risk prediction models for discrete ordinal outcomes: calibration and the impact of the proportional odds assumption

04/19/2021
by   Michael Edlinger, et al.
0

Calibration is a vital aspect of the performance of risk prediction models, but research in the context of ordinal outcomes is scarce. This study compared calibration measures for risk models predicting a discrete ordinal outcome, and investigated the impact of the proportional odds assumption on calibration and overfitting. We studied the multinomial, cumulative, adjacent category, continuation ratio, and stereotype logit/logistic models. To assess calibration, we investigated calibration intercepts and slopes, calibration plots, and the estimated calibration index. Using large sample simulations, we studied the performance of models for risk estimation under various conditions, assuming that the true model has either a multinomial logistic form or a cumulative logit proportional odds form. Small sample simulations were used to compare the tendency for overfitting between models. As a case study, we developed models to diagnose the degree of coronary artery disease (five categories) in symptomatic patients. When the true model was multinomial logistic, proportional odds models often yielded poor risk estimates, with calibration slopes deviating considerably from unity even on large model development datasets. The stereotype logistic model improved the calibration slope, but still provided biased risk estimates for individual patients. When the true model had a cumulative logit proportional odds form, multinomial logistic regression provided biased risk estimates, although these biases were modest. Non-proportional odds models, however, required more parameters to be estimated from the data, and hence suffered more from overfitting. Despite larger sample size requirements, we generally recommend multinomial logistic regression for risk prediction modeling of discrete ordinal outcomes.

READ FULL TEXT

page 34

page 35

page 36

07/26/2022

Minimum Sample Size for Developing a Multivariable Prediction Model using Multinomial Logistic Regression

Multinomial logistic regression models allow one to predict the risk of ...
02/17/2022

AutoScore-Ordinal: An interpretable machine learning framework for generating scoring models for ordinal outcomes

Background: Risk prediction models are useful tools in clinical decision...
12/25/2010

Ordinal Risk-Group Classification

Most classification methods provide either a prediction of class members...
10/13/2021

Bayesian logistic regression for online recalibration and revision of risk prediction models with performance guarantees

After deploying a clinical prediction model, subsequently collected data...
01/07/2021

Kullback-Leibler-Based Discrete Relative Risk Models for Integration of Published Prediction Models with New Dataset

Existing literature for prediction of time-to-event data has primarily f...
01/30/2020

Assessing the Calibration of Subdistribution Hazard Models in Discrete Time

The generalization performance of a risk prediction model can be evaluat...
10/16/2021

Minding non-collapsibility of odds ratios when recalibrating risk prediction models

In clinical prediction modeling, model updating refers to the practice o...