DeepAI AI Chat
Log In Sign Up

Percentile-Based Residuals for Model Assessment

by   Sophie Bérubé, et al.
Johns Hopkins Medicine

Residuals are a key component of diagnosing model fit. The usual practice is to compute standardized residuals using expected values and standard deviations of the observed data, then use these values to detect outliers and assess model fit. Approximate normality of these residuals is key for this process to have good properties, but in many modeling contexts, especially for complex, multi-level models, normality may not hold. In these cases outlier detection and model diagnostics aren't properly calibrated. Alternatively, as we demonstrate, residuals computed from the percentile location of a datum's value in its full predictive distribution lead to well calibrated evaluations of model fit. We generalize an approach described by Dunn and Smyth (1996) and evaluate properties mathematically, via case-studies and by simulation. In addition, we show that the standard residuals can be calibrated to mimic the percentile approach, but that this extra step is avoided by directly using percentile-based residuals. For both the percentile-based residuals and the calibrated standard residuals, the use of full predictive distributions with the appropriate location, spread and shape is necessary for valid assessments.


page 1

page 2

page 3

page 4


Calibrating generalized predictive distributions

In prediction problems, it is common to model the data-generating proces...

Strategic model reduction by analysing model sloppiness: a case study in coral calcification

It can be difficult to identify ways to reduce the complexity of large m...

Interpolation of Missing Swaption Volatility Data using Gibbs Sampling on Variational Autoencoders

Albeit of crucial interest for both financial practitioners and research...

Characterization and valuation of uncertainty of calibrated parameters in stochastic decision models

We evaluated the implications of different approaches to characterize un...

Calibrated Adversarial Refinement for Multimodal Semantic Segmentation

Ambiguities in images or unsystematic annotation can lead to multiple va...

A Claim Score for Dynamic Claim Counts Modeling

We develop a claim score based on the Bonus-Malus approach proposed by [...

Conformal calibrators

Most existing examples of full conformal predictive systems, split-confo...