Evaluating the Discrimination Ability of Proper Multivariate Scoring Rules

by   Carol Alexander, et al.

Proper scoring rules are commonly applied to quantify the accuracy of distribution forecasts. Given an observation they assign a scalar score to each distribution forecast, with the the lowest expected score attributed to the true distribution. The energy and variogram scores are two rules that have recently gained some popularity in multivariate settings because their computation does not require a forecast to have parametric density function and so they are broadly applicable. Here we conduct a simulation study to compare the discrimination ability between the energy score and three variogram scores. Compared with other studies, our simulation design is more realistic because it is supported by a historical data set containing commodity prices, currencies and interest rates, and our data generating processes include a diverse selection of models with different marginal distributions, dependence structure, and calibration windows. This facilitates a comprehensive comparison of the performance of proper scoring rules in different settings. To compare the scores we use three metrics: the mean relative score, error rate and a generalised discrimination heuristic. Overall, we find that the variogram score with parameter p=0.5 outperforms the energy score and the other two variogram scores.



There are no comments yet.


page 1

page 2

page 3

page 4


Validation of point process predictions with proper scoring rules

We introduce a class of proper scoring rules for evaluating spatial poin...

Scoring Functions for Multivariate Distributions and Level Sets

Interest in predicting multivariate probability distributions is growing...

Scoring Alternative Forecast Distributions: Completing the Kullback Distance Complex

We develop two surprising new results regarding the use of proper scorin...

Threshold Choice Methods: the Missing Link

Many performance metrics have been introduced for the evaluation of clas...

Proper scoring rules for evaluating asymmetry in density forecasting

This paper proposes a novel asymmetric continuous probabilistic score (A...

Random Noise vs State-of-the-Art Probabilistic Forecasting Methods : A Case Study on CRPS-Sum Discrimination Ability

The recent developments in the machine learning domain have enabled the ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.