DeepAI AI Chat
Log In Sign Up

Evaluating probabilistic forecasts of football matches: The case against the Ranked Probability Score

by   Edward Wheatcroft, et al.

A scoring rule is a function of a probabilistic forecast and a corresponding outcome that is used to evaluate forecast performance. A wide range of scoring rules have been defined over time and there is some debate as to which are the most appropriate for evaluating the performance of forecasts of sporting events. This paper focuses on forecasts of the outcomes of football matches. The ranked probability score (RPS) is often recommended since it is `sensitive to distance', that is it takes into account the ordering in the outcomes (a home win is `closer' to a draw than it is to an away win, for example). In this paper, this reasoning is disputed on the basis that it adds nothing in terms of the actual aims of using scoring rules. A related property of scoring rules is locality. A scoring rule is local if it only takes the probability placed on the outcome into consideration. Two simulation experiments are carried out in the context of football matches to compare the performance of the RPS, which is non-local and sensitive to distance, the Brier score, which is non-local and insensitive to distance, and the ignorance score, which is local and insensitive to distance. The ignorance score is found to outperform both the RPS and the Brier score, casting doubt on the value of non-locality and sensitivity to distance as properties of scoring rules in this context.


page 11

page 12


Weighted scoringRules: Emphasising Particular Outcomes when Evaluating Probabilistic Forecasts

When predicting future events, it is common to issue forecasts that are ...

Retrodictive Modelling of Modern Rugby Union: Extension of Bradley-Terry to Multiple Outcomes

Frequently in sporting competitions it is desirable to compare teams bas...

Mathematical Properties of Continuous Ranked Probability Score Forecasting

The theoretical advances on the properties of scoring rules over the pas...

Evaluating forecasts for high-impact events using transformed kernel scores

It is informative to evaluate a forecaster's ability to predict outcomes...

Online Aggregation of Probability Forecasts with Confidence

The paper presents numerical experiments and some theoretical developmen...

Wisdom of the crowds forecasting the 2018 FIFA Men's World Cup

The FIFA Men's World Cup Tournament (WCT) is the most important football...