On the variability of regression shrinkage methods for clinical prediction models: simulation study on predictive performance

07/26/2019
by   Ben van Calster, et al.
0

When developing risk prediction models, shrinkage methods are recommended, especially when the sample size is limited. Several earlier studies have shown that the shrinkage of model coefficients can reduce overfitting of the prediction model and subsequently result in better predictive performance on average. In this simulation study, we aimed to investigate the variability of regression shrinkage on predictive performance for a binary outcome, with focus on the calibration slope. The slope indicates whether risk predictions are too extreme (slope < 1) or not extreme enough (slope > 1). We investigated the following shrinkage methods in comparison to standard maximum likelihood estimation: uniform shrinkage (likelihood-based and bootstrap-based), ridge regression, penalized maximum likelihood, LASSO regression, adaptive LASSO, non-negative garrote, and Firth's correction. There were three main findings. First, shrinkage improved calibration slopes on average. Second, the between-sample variability of calibration slopes was often increased relative to maximum likelihood. Among the shrinkage methods, the bootstrap-based uniform shrinkage worked well overall. In contrast to other shrinkage approaches, Firth's correction had only a small shrinkage effect but did so with low variability. Third, the correlation between the estimated shrinkage and the optimal shrinkage to remove overfitting was typically negative. Hence, although shrinkage improved predictions on average, it often worked poorly in individual datasets, in particular when shrinkage was most needed. The observed variability of shrinkage methods implies that these methods do not solve problems associated with small sample size or low number of events per variable.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

01/27/2021

To tune or not to tune, a case study of ridge logistic regression in small or sparse datasets

For finite samples with binary outcomes penalized logistic regression su...
10/17/2018

Shrinkage estimation of rate statistics

This paper presents a simple shrinkage estimator of rates based on Bayes...
09/12/2018

Prediction out-of-sample using block shrinkage estimators: model selection and predictive inference

In a linear regression model with random design, we consider a family of...
07/24/2018

Improving pairwise comparison models using Empirical Bayes shrinkage

Comparison data arises in many important contexts, e.g. shopping, web cl...
12/09/2019

Semiparametric Regression for Dual Population Mortality

Parameter shrinkage applied optimally can always reduce error and projec...
03/09/2021

The Efficient Shrinkage Path: Maximum Likelihood of Minimum MSE Risk

A new generalized ridge regression shrinkage path is proposed that is as...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.