Valid sequential inference on probability forecast performance

03/15/2021
by   Alexander Henzi, et al.
0

Probability forecasts for binary events play a central role in many applications. Their quality is commonly assessed with proper scoring rules, which assign forecasts a numerical score such that a correct forecast achieves a minimal expected score. In this paper, we construct e-values for testing the statistical significance of score differences of competing forecasts in sequential settings. E-values have been proposed as an alternative to p-values for hypothesis testing, and they can easily be transformed into conservative p-values by taking the multiplicative inverse. The e-values proposed in this article are valid in finite samples without any assumptions on the data generating processes. They also allow optional stopping, so a forecast user may decide to interrupt evaluation taking into account the available data at any time and still draw statistically valid inference, which is generally not true for classical p-value based tests. In a case study on postprocessing of precipitation forecasts, state-of-the-art forecasts dominance tests and e-values lead to the same conclusions.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 22

09/24/2021

Sequentially valid tests for forecast calibration

Forecasting and forecast evaluation are inherently sequential tasks. Pre...
09/30/2021

Comparing Sequential Forecasters

Consider two or more forecasters, each making a sequence of predictions ...
05/25/2021

Ranking earthquake forecasts using proper scoring rules: Binary events in a low probability environment

Operational earthquake forecasting for risk management and communication...
02/25/2022

Evaluating forecasts for high-impact events using transformed kernel scores

It is informative to evaluate a forecaster's ability to predict outcomes...
04/10/2020

Forecasts with Bayesian vector autoregressions under real time conditions

This paper investigates the sensitivity of forecast performance measures...
10/28/2019

Testing Forecast Rationality for Measures of Central Tendency

Rational respondents to economic surveys may report as a point forecast ...
09/06/2021

Using Proxies to Improve Forecast Evaluation

Comparative evaluation of forecasts of statistical functionals relies on...

Code Repositories

eprob

E-values for sequential inference on probability forecast performance


view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.