Uncertainty and Value of Perfect Information in Risk Prediction Modeling

by   Mohsen Sadatsafavi, et al.

Background: Predicted probabilities from a risk prediction model are inevitably uncertain. This uncertainty has mostly been studied from a statistical perspective. We apply Value of Information methodology to evaluate the decision-theoretic implications of prediction uncertainty. Methods: Adopting a Bayesian perspective, we extend the definition of the Expected Value of Perfect Information (EVPI) from decision analysis to net benefit calculations in risk prediction. EVPI is the expected gain in net benefit by using the correct predictions as opposed to predictions from a proposed model. We suggest bootstrap methods for sampling from the posterior distribution of predictions for EVPI calculation using Monte Carlo simulations. In a case study, we used subsets of data of various sizes from a clinical trial for predicting mortality after myocardial infarction to show how EVPI can be interpreted and how it changes with sample size. Results: With a sample size of 1,000, EVPI was 0 at threshold values larger than 0.6, indicating there is no point in procuring more development data for such thresholds. At thresholds of 0.4-0.6, the proposed model was not net beneficial, but EVPI was positive, indicating that obtaining more development data might be justified. Across all thresholds, the gain in net benefit by using the correct model was 24 model. EVPI declined with larger samples and was generally low with sample sizes of 4,000 or greater. We summarize an algorithm for incorporating EVPI calculations into the commonly used bootstrap method for optimism correction. Conclusion: Value of Information methods can be applied to explore decision-theoretic consequences of uncertainty in risk prediction, and can complement inferential methods when developing or validating risk prediction models.


Value of Information Analysis for External Validation of Risk Prediction Models

Background: Before being used to inform patient care, a risk prediction ...

Decision curve analysis for personalized treatment choice between multiple options

Decision curve analysis can be used to determine whether a personalized ...

Estimating the Expected Value of Sample Information across Different Sample Sizes using Moment Matching and Non-Linear Regression

Background: The Expected Value of Sample Information (EVSI) determines t...

The use of restricted mean time lost under competing risks data

Background: Under competing risks, the commonly used sub-distribution ha...

Decision Tree Learning for Uncertain Clinical Measurements

Clinical decision requires reasoning in the presence of imperfect data. ...