Risk prediction models for discrete ordinal outcomes: calibration and the impact of the proportional odds assumption

by   Michael Edlinger, et al.

Calibration is a vital aspect of the performance of risk prediction models, but research in the context of ordinal outcomes is scarce. This study compared calibration measures for risk models predicting a discrete ordinal outcome, and investigated the impact of the proportional odds assumption on calibration and overfitting. We studied the multinomial, cumulative, adjacent category, continuation ratio, and stereotype logit/logistic models. To assess calibration, we investigated calibration intercepts and slopes, calibration plots, and the estimated calibration index. Using large sample simulations, we studied the performance of models for risk estimation under various conditions, assuming that the true model has either a multinomial logistic form or a cumulative logit proportional odds form. Small sample simulations were used to compare the tendency for overfitting between models. As a case study, we developed models to diagnose the degree of coronary artery disease (five categories) in symptomatic patients. When the true model was multinomial logistic, proportional odds models often yielded poor risk estimates, with calibration slopes deviating considerably from unity even on large model development datasets. The stereotype logistic model improved the calibration slope, but still provided biased risk estimates for individual patients. When the true model had a cumulative logit proportional odds form, multinomial logistic regression provided biased risk estimates, although these biases were modest. Non-proportional odds models, however, required more parameters to be estimated from the data, and hence suffered more from overfitting. Despite larger sample size requirements, we generally recommend multinomial logistic regression for risk prediction modeling of discrete ordinal outcomes.


page 34

page 35

page 36


Minimum Sample Size for Developing a Multivariable Prediction Model using Multinomial Logistic Regression

Multinomial logistic regression models allow one to predict the risk of ...

AutoScore-Ordinal: An interpretable machine learning framework for generating scoring models for ordinal outcomes

Background: Risk prediction models are useful tools in clinical decision...

Ordinal Risk-Group Classification

Most classification methods provide either a prediction of class members...

Bayesian logistic regression for online recalibration and revision of risk prediction models with performance guarantees

After deploying a clinical prediction model, subsequently collected data...

Kullback-Leibler-Based Discrete Relative Risk Models for Integration of Published Prediction Models with New Dataset

Existing literature for prediction of time-to-event data has primarily f...

Assessing the Calibration of Subdistribution Hazard Models in Discrete Time

The generalization performance of a risk prediction model can be evaluat...

Minding non-collapsibility of odds ratios when recalibrating risk prediction models

In clinical prediction modeling, model updating refers to the practice o...