Random Noise vs State-of-the-Art Probabilistic Forecasting Methods : A Case Study on CRPS-Sum Discrimination Ability

by   Alireza Koochali, et al.
Technische Universität Kaiserslautern

The recent developments in the machine learning domain have enabled the development of complex multivariate probabilistic forecasting models. Therefore, it is pivotal to have a precise evaluation method to gauge the performance and predictability power of these complex methods. To do so, several evaluation metrics have been proposed in the past (such as Energy Score, Dawid-Sebastiani score, variogram score), however, they cannot reliably measure the performance of a probabilistic forecaster. Recently, CRPS-sum has gained a lot of prominence as a reliable metric for multivariate probabilistic forecasting. This paper presents a systematic evaluation of CRPS-sum to understand its discrimination ability. We show that the statistical properties of target data affect the discrimination ability of CRPS-Sum. Furthermore, we highlight that CRPS-Sum calculation overlooks the performance of the model on each dimension. These flaws can lead us to an incorrect assessment of model performance. Finally, with experiments on the real-world dataset, we demonstrate that the shortcomings of CRPS-Sum provide a misleading indication of the probabilistic forecasting performance method. We show that it is easily possible to have a better CRPS-Sum for a dummy model, which looks like random noise, in comparison to the state-of-the-art method.


Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting

In this work, we propose TimeGrad, an autoregressive model for multivari...

Regions of Reliability in the Evaluation of Multivariate Probabilistic Forecasts

Multivariate probabilistic time series forecasts are commonly evaluated ...

Probabilistic Forecasting of Sensory Data with Generative Adversarial Networks - ForGAN

Time series forecasting is one of the challenging problems for humankind...

Quantile Surfaces – Generalizing Quantile Regression to Multivariate Targets

In this article, we present a novel approach to multivariate probabilist...

A copula-based time series model for global horizontal irradiation

The increasing importance of solar power for electricity generation lead...

Parameter Efficient Deep Probabilistic Forecasting

Probabilistic time series forecasting is crucial in many application dom...

If You Like It, GAN It. Probabilistic Multivariate Times Series Forecast With GAN

The contribution of this paper is two-fold. First, we present ProbCast -...

Please sign up or login with your details

Forgot password? Click here to reset