
Optimization of Scoring Rules
This paper introduces an objective for optimizing proper scoring rules. ...
read it

A new example for a proper scoring rule
We give a new example for a proper scoring rule motivated by the form of...
read it

Objective Bayesian inference with proper scoring rules
Standard Bayesian analyses can be difficult to perform when the full lik...
read it

Shift Happens: Adjusting Classifiers
Minimizing expected loss measured by a proper scoring rule, such as Brie...
read it

Alignment Problems With Current Forecasting Platforms
We present alignment problems in current forecasting platforms, such as ...
read it

Binary Scoring Rules that Incentivize Precision
All proper scoring rules incentivize an expert to predict accurately (re...
read it

Eliciting Social Knowledge for Creditworthiness Assessment
Access to capital is a major constraint for economic growth in the devel...
read it
Optimal Scoring Rule Design
This paper introduces an optimization problem for proper scoring rule design. Consider a principal who wants to collect an agent's prediction about an unknown state. The agent can either report his prior prediction or access a costly signal and report the posterior prediction. Given a collection of possible distributions containing the agent's posterior prediction distribution, the principal's objective is to design a bounded scoring rule to maximize the agent's worstcase payoff increment between reporting his posterior prediction and reporting his prior prediction. We study two settings of such optimization for proper scoring rules: static and asymptotic settings. In the static setting, where the agent can access one signal, we propose an efficient algorithm to compute an optimal scoring rule when the collection of distributions is finite. The agent can adaptively and indefinitely refine his prediction in the asymptotic setting. We first consider a sequence of collections of posterior distributions with vanishing covariance, which emulates general estimators with large samples, and show the optimality of the quadratic scoring rule. Then, when the agent's posterior distribution is a BetaBernoulli process, we find that the log scoring rule is optimal. We also prove the optimality of the log scoring rule over a smaller set of functions for categorical distributions with Dirichlet priors.
READ FULL TEXT
Comments
There are no comments yet.