Bayesian logistic regression for online recalibration and revision of risk prediction models with performance guarantees

10/13/2021
by   Jean Feng, et al.
0

After deploying a clinical prediction model, subsequently collected data can be used to fine-tune its predictions and adapt to temporal shifts. Because model updating carries risks of over-updating/fitting, we study online methods with performance guarantees. We introduce two procedures for continual recalibration or revision of an underlying prediction model: Bayesian logistic regression (BLR) and a Markov variant that explicitly models distribution shifts (MarBLR). We perform empirical evaluation via simulations and a real-world study predicting COPD risk. We derive "Type I and II" regret bounds, which guarantee the procedures are non-inferior to a static model and competitive with an oracle logistic reviser in terms of the average loss. Both procedures consistently outperformed the static model and other online logistic revision methods. In simulations, the average estimated calibration index (aECI) of the original model was 0.828 (95 recalibration using BLR and MarBLR improved the aECI, attaining 0.265 (95 0.230-0.300) and 0.241 (95 extensive logistic model revisions, BLR and MarBLR increased the average AUC (aAUC) from 0.767 (95 (95 substantial model decay. In the COPD study, BLR and MarBLR dynamically combined the original model with a continually-refitted gradient boosted tree to achieve aAUCs of 0.924 (95 the static model's aAUC of 0.904 (95 BLR is highly competitive with MarBLR. MarBLR outperforms BLR when its prior better reflects the data. BLR and MarBLR can improve the transportability of clinical prediction models and maintain their performance over time.

READ FULL TEXT

page 1

page 2

page 3

page 4

10/16/2021

Minding non-collapsibility of odds ratios when recalibrating risk prediction models

In clinical prediction modeling, model updating refers to the practice o...
06/14/2017

Predictive modelling of training loads and injury in Australian football

To investigate whether training load monitoring data could be used to pr...
04/19/2021

Risk prediction models for discrete ordinal outcomes: calibration and the impact of the proportional odds assumption

Calibration is a vital aspect of the performance of risk prediction mode...
01/21/2020

Clinical Prediction Models to Predict the Risk of Multiple Binary Outcomes: a comparison of approaches

Clinical prediction models (CPMs) are used to predict clinically relevan...
02/26/2019

Continual Prediction from EHR Data for Inpatient Acute Kidney Injury

Acute kidney injury (AKI) commonly occurs in hospitalized patients and c...
10/22/2020

Model updating after interventions paradoxically introduces bias

Machine learning is increasingly being used to generate prediction model...
03/30/2022

Prognosis of Rotor Parts Fly-off Based on Cascade Classification and Online Prediction Ability Index

Large rotating machines, e.g., compressors, steam turbines, gas turbines...