Confidence intervals of prediction accuracy measures for multivariable prediction models based on the bootstrap-based optimism correction methods

05/04/2020
by   Hisashi Noma, et al.
0

In assessing prediction accuracy of multivariable prediction models, optimism corrections are essential for preventing biased results. However, in most published papers of clinical prediction models, the point estimates of the prediction accuracy measures are corrected by adequate bootstrap-based correction methods, but their confidence intervals are not corrected, e.g., the DeLong's confidence interval is usually used for assessing the C-statistic. These naive methods do not adjust for the optimism bias and do not account for statistical variability in the estimation of parameters in the prediction models. Therefore, their coverage probabilities of the true value of the prediction accuracy measure can be seriously below the nominal level (e.g., 95 location-shifted bootstrap confidence intervals and (2) two-stage bootstrap confidence intervals, that can be generally applied to the bootstrap-based optimism correction methods, i.e., the Harrell's bias correction, 0.632, and 0.632+ methods. In addition, they can be widely applied to various methods for prediction model development involving modern shrinkage methods such as the ridge and lasso regressions. Through numerical evaluations by simulations, the proposed confidence intervals showed favourable coverage performances. Besides, the current standard practices based on the optimism-uncorrected methods showed serious undercoverage properties. To avoid erroneous results, the optimism-uncorrected confidence intervals should not be used in practice, and the adjusted methods are recommended instead. We also developed the R package predboot for implementing these methods (https://github.com/nomahi/predboot). The effectiveness of the proposed methods are illustrated via applications to the GUSTO-I clinical trial.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/26/2022

Confidence Intervals for the Generalisation Error of Random Forests

Out-of-bag error is commonly used as an estimate of generalisation error...
research
01/24/2023

Think before you shrink: Alternatives to default shrinkage methods can improve prediction accuracy, calibration and coverage

While shrinkage is essential in high-dimensional settings, its use for l...
research
06/11/2021

Bootstrapping Clustered Data in R using lmeresampler

Linear mixed-effects models are commonly used to analyze clustered data ...
research
07/26/2022

Risk-Adjusted Incidence Modeling on Hierarchical Survival Data with Recurrent Events

There is a constant need for many healthcare programs to timely address ...
research
05/23/2022

Please, Don't Forget the Difference and the Confidence Interval when Seeking for the State-of-the-Art Status

This paper argues for the widest possible use of bootstrap confidence in...
research
04/21/2022

New confidence interval methods for Shannon index

Several factors affect the structure of communities, including biologica...

Please sign up or login with your details

Forgot password? Click here to reset