DeepAI AI Chat
Log In Sign Up

Evaluating individualized treatment effect predictions: a new perspective on discrimination and calibration assessment

by   J Hoogland, et al.

Personalized medicine constitutes a growing area of research that benefits from the many new developments in statistical learning. A key domain concerns the prediction of individualized treatment effects, and models for this purpose are increasingly common in the published literature. Aiming to facilitate the validation of prediction models for individualized treatment effects, we extend the classical concepts of discrimination and calibration performance to assess causal (rather than associative) prediction models. Working within the potential outcomes framework, we first evaluate properties of existing statistics (including the c-for-benefit) and subsequently propose novel model-based statistics. The main focus is on randomized trials with binary endpoints. We use simulated data to provide insight into the characteristics of discrimination and calibration statistics, and further illustrate all methods in a trial in acute ischemic stroke treatment. Results demonstrate that the proposed model-based statistics had the best characteristics in terms of bias and variance. While resampling methods to adjust for optimism of performance estimates in the development data were effective on average, they had a high variance across replications that limits their accuracy in any particular applied analysis. Thereto, individualized treatment effect models are best validated in external data rather than in the original development sample.


page 1

page 2

page 3

page 4


A tutorial on individualized treatment effect prediction from randomized trials with a binary endpoint

Randomized trials typically estimate average relative treatment effects,...

Prediction under hypothetical interventions: evaluation of performance using longitudinal observational data

Prediction models provide risks of an adverse event occurring for an ind...

Model-based ROC (mROC) curve: examining the effect of case-mix and model calibration on the ROC plot

The performance of a risk prediction model is often characterized in ter...

Calibration Error for Heterogeneous Treatment Effects

Recently, many researchers have advanced data-driven methods for modelin...

A Framework for the Meta-Analysis of Randomized Experiments with Applications to Heavy-Tailed Response Data

A central obstacle in the objective assessment of treatment effect (TE) ...