Forecast Evaluation of Set-Valued Functionals

10/16/2019
by   Tobias Fissler, et al.
0

A functional is elicitable (identifiable) if it is the unique minimiser (zero) of an expected scoring function (identification function). Elicitability and identifiability are essential for forecast ranking and validation, M- and Z-estimation, both possibly in a regression framework. To account for the set-valued nature of many interesting functionals such as quantiles, systemic risk measures or prediction intervals we introduce a theoretical framework of elicitability and identifiability of set-valued functionals. It distinguishes between exhaustive forecasts, being set-valued and aiming at correctly specifying the entire functional, and selective forecasts, content with solely specifying a single point in the correct functional. Uncovering the structural relation between the two corresponding notions of elicitability and identifiability, we establish that a set-valued functional can be either selectively elicitable or exhaustively elicitable. Notably, selections of quantiles such as the lower quantile turn out not to be elicitable in general. Applying these structural results to Vorob'ev quantiles of random sets, we establish their selective identifiability and exhaustive elicitability. In particular, we provide a mixture representation of elementary scores, leading the way to Murphy diagrams. Our paper is complemented by a comprehensive literature review elaborating on common practice in forecast evaluation of set-valued functionals.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/02/2019

Elicitability and Identifiability of Systemic Risk Measures and other Set-Valued Functionals

This paper is concerned with a two-fold objective. Firstly, we establish...
research
03/16/2022

Measurability of functionals and of ideal point forecasts

The ideal probabilistic forecast for a random variable Y based on an inf...
research
08/16/2022

Osband's Principle for Identification Functions

Given a statistical functional of interest such as the mean or median, a...
research
08/27/2021

Point forecasting and forecast evaluation with generalized Huber loss

Huber loss, its asymmetric variants and their associated functionals (he...
research
08/06/2021

Regression Diagnostics meets Forecast Evaluation: Conditional Calibration, Reliability Diagrams, and Coefficient of Determination

Model diagnostics and forecast evaluation are two sides of the same coin...
research
12/15/2019

Integral Mixabilty: a Tool for Efficient Online Aggregation of Functional and Probabilistic Forecasts

In this paper we extend the setting of the online prediction with expert...
research
07/11/2020

Scoring Interval Forecasts: Equal-Tailed, Shortest, and Modal Interval

We consider different types of predictive intervals and ask whether they...

Please sign up or login with your details

Forgot password? Click here to reset