Valid sequential inference on probability forecast performance

by   Alexander Henzi, et al.

Probability forecasts for binary events play a central role in many applications. Their quality is commonly assessed with proper scoring rules, which assign forecasts a numerical score such that a correct forecast achieves a minimal expected score. In this paper, we construct e-values for testing the statistical significance of score differences of competing forecasts in sequential settings. E-values have been proposed as an alternative to p-values for hypothesis testing, and they can easily be transformed into conservative p-values by taking the multiplicative inverse. The e-values proposed in this article are valid in finite samples without any assumptions on the data generating processes. They also allow optional stopping, so a forecast user may decide to interrupt evaluation taking into account the available data at any time and still draw statistically valid inference, which is generally not true for classical p-value based tests. In a case study on postprocessing of precipitation forecasts, state-of-the-art forecasts dominance tests and e-values lead to the same conclusions.



There are no comments yet.


page 22


Sequentially valid tests for forecast calibration

Forecasting and forecast evaluation are inherently sequential tasks. Pre...

Comparing Sequential Forecasters

Consider two or more forecasters, each making a sequence of predictions ...

Ranking earthquake forecasts using proper scoring rules: Binary events in a low probability environment

Operational earthquake forecasting for risk management and communication...

Evaluating forecasts for high-impact events using transformed kernel scores

It is informative to evaluate a forecaster's ability to predict outcomes...

Forecasts with Bayesian vector autoregressions under real time conditions

This paper investigates the sensitivity of forecast performance measures...

Testing Forecast Rationality for Measures of Central Tendency

Rational respondents to economic surveys may report as a point forecast ...

Using Proxies to Improve Forecast Evaluation

Comparative evaluation of forecasts of statistical functionals relies on...

Code Repositories


E-values for sequential inference on probability forecast performance

view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.