Binary Scoring Rules that Incentivize Precision

02/25/2020
by   Eric Neyman, et al.
0

All proper scoring rules incentivize an expert to predict accurately (report their true estimate), but not all proper scoring rules equally incentivize precision. Rather than consider the expert's belief as exogenously given, we consider a model where a rational expert can endogenously refine their belief by repeatedly paying a fixed cost, and is incentivized to do so by a proper scoring rule. Specifically, our expert aims to predict the probability that a biased coin flipped tomorrow will land heads or tails, and can flip the coin any number of times today at a cost of c per flip. Our first main result defines an incentivization index for proper scoring rules, and proves that this index measures the expected error of the expert's estimate (where the number of flips today is chosen adaptively to maximize the predictor's expected payoff). Our second main result finds the unique scoring rule which optimizes the incentivization index over all proper scoring rules. We also consider extensions to minimizing the ℓ^th moment of error, and again provide an incentivization index and optimal proper scoring rule. In some cases, the resulting scoring rule is differentiable, but not infinitely differentiable. In these cases, we further prove that the optimum can be uniformly approximated by polynomial scoring rules. Finally, we compare common scoring rules via our measure, and include simulations confirming the relevance of our measure even in domains outside where it provably applies.

READ FULL TEXT
research
07/06/2020

Optimization of Scoring Rules

This paper introduces an objective for optimizing proper scoring rules. ...
research
12/16/2019

A new example for a proper scoring rule

We give a new example for a proper scoring rule motivated by the form of...
research
01/07/2014

Cortical prediction markets

We investigate cortical learning from the perspective of mechanism desig...
research
06/19/2018

Properization

Scoring rules serve to quantify predictive performance. A scoring rule i...
research
06/19/2018

Properization: Constructing Proper Scoring Rules via Bayes Acts

Scoring rules serve to quantify predictive performance. A scoring rule i...
research
07/28/2022

Model selection with Gini indices under auto-calibration

The Gini index does not give a strictly consistent scoring rule in gener...
research
05/25/2018

In Design DFM Rule Scoring and Fixing Method using ICV

As compared to DRC rules, DFM rules are a list of selected recommended r...

Please sign up or login with your details

Forgot password? Click here to reset