DeepAI AI Chat
Log In Sign Up

Model Comparison and Calibration Assessment: User Guide for Consistent Scoring Functions in Machine Learning and Actuarial Practice

by   Tobias Fissler, et al.

One of the main tasks of actuaries and data scientists is to build good predictive models for certain phenomena such as the claim size or the number of claims in insurance. These models ideally exploit given feature information to enhance the accuracy of prediction. This user guide revisits and clarifies statistical techniques to assess the calibration or adequacy of a model on the one hand, and to compare and rank different models on the other hand. In doing so, it emphasises the importance of specifying the prediction target at hand a priori and of choosing the scoring function in model comparison in line with this target. Guidance for the practical choice of the scoring function is provided. Striving to bridge the gap between science and daily practice in application, it focuses mainly on the pedagogical presentation of existing results and of best practice. The results are accompanied and illustrated by two real data case studies on workers' compensation and customer churn.


page 4

page 24


Development and Comparison of Scoring Functions in Curriculum Learning

Curriculum Learning is the presentation of samples to the machine learni...

Credit scoring using neural networks and SURE posterior probability calibration

In this article we compare the performances of a logistic regression and...

Calibration Scoring Rules for Practical Prediction Training

In situations where forecasters are scored on the quality of their proba...

Using scoring functions to evaluate point process forecasts

Point process models are widely used tools to issue forecasts or assess ...

Model selection with Gini indices under auto-calibration

The Gini index does not give a strictly consistent scoring rule in gener...

Scoring Predictions at Extreme Quantiles

Prediction of quantiles at extreme tails is of interest in numerous appl...

Competitive Machine Learning: Best Theoretical Prediction vs Optimization

Machine learning is often used in competitive scenarios: Participants le...