On the variability of regression shrinkage methods for clinical prediction models: simulation study on predictive performance

by   Ben van Calster, et al.

When developing risk prediction models, shrinkage methods are recommended, especially when the sample size is limited. Several earlier studies have shown that the shrinkage of model coefficients can reduce overfitting of the prediction model and subsequently result in better predictive performance on average. In this simulation study, we aimed to investigate the variability of regression shrinkage on predictive performance for a binary outcome, with focus on the calibration slope. The slope indicates whether risk predictions are too extreme (slope < 1) or not extreme enough (slope > 1). We investigated the following shrinkage methods in comparison to standard maximum likelihood estimation: uniform shrinkage (likelihood-based and bootstrap-based), ridge regression, penalized maximum likelihood, LASSO regression, adaptive LASSO, non-negative garrote, and Firth's correction. There were three main findings. First, shrinkage improved calibration slopes on average. Second, the between-sample variability of calibration slopes was often increased relative to maximum likelihood. Among the shrinkage methods, the bootstrap-based uniform shrinkage worked well overall. In contrast to other shrinkage approaches, Firth's correction had only a small shrinkage effect but did so with low variability. Third, the correlation between the estimated shrinkage and the optimal shrinkage to remove overfitting was typically negative. Hence, although shrinkage improved predictions on average, it often worked poorly in individual datasets, in particular when shrinkage was most needed. The observed variability of shrinkage methods implies that these methods do not solve problems associated with small sample size or low number of events per variable.



There are no comments yet.


page 1

page 2

page 3

page 4


To tune or not to tune, a case study of ridge logistic regression in small or sparse datasets

For finite samples with binary outcomes penalized logistic regression su...

Shrinkage estimation of rate statistics

This paper presents a simple shrinkage estimator of rates based on Bayes...

Prediction out-of-sample using block shrinkage estimators: model selection and predictive inference

In a linear regression model with random design, we consider a family of...

Improving pairwise comparison models using Empirical Bayes shrinkage

Comparison data arises in many important contexts, e.g. shopping, web cl...

Semiparametric Regression for Dual Population Mortality

Parameter shrinkage applied optimally can always reduce error and projec...

The Efficient Shrinkage Path: Maximum Likelihood of Minimum MSE Risk

A new generalized ridge regression shrinkage path is proposed that is as...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.