Forecast Aggregation via Peer Prediction

10/09/2019
by   Juntao Wang, et al.
0

Crowdsourcing is a popular paradigm for soliciting forecasts on future events. As people may have different forecasts, how to aggregate solicited forecasts into a single accurate prediction remains to be an important challenge, especially when no historical accuracy information is available for identifying experts. In this paper, we borrow ideas from the peer prediction literature and assess the prediction accuracy of participants using solely the collected forecasts. This approach leverages the correlations among peer reports to cross-validate each participant's forecasts and allows us to assign a "peer assessment score (PAS)" for each agent as a proxy for the agent's prediction accuracy. We identify several empirically effective methods to generate PAS and propose an aggregation framework that uses PAS to identify experts and to boost existing aggregators' prediction accuracy. We evaluate our methods over 14 real-world datasets and show that i) PAS generated from peer prediction methods can approximately reflect the prediction accuracy of agents, and ii) our aggregation framework demonstrates consistent and significant improvement in the prediction accuracy over existing aggregators for both binary and multi-choice questions under three popular accuracy measures: Brier score (mean square error), log score (cross-entropy loss) and AUC-ROC.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/20/2018

Learning of Optimal Forecast Aggregation in Partial Evidence Environments

We consider the forecast aggregation problem in repeated settings, where...
research
04/18/2023

Probabilistic Forecast-based Portfolio Optimization of Electricity Demand at Low Aggregation Levels

In the effort to achieve carbon neutrality through a decentralized elect...
research
03/20/2023

Proxy Forecasting to Avoid Stochastic Decision Rules in Decision Markets

Information that is of relevance for decision-making is often distribute...
research
03/03/2020

Anchor Attention for Hybrid Crowd Forecasts Aggregation

Forecasting the future is a notoriously difficult task. To overcome this...
research
07/03/2019

Sequence to Sequence with Attention for Influenza Prevalence Prediction using Google Trends

Early prediction of the prevalence of influenza reduces its impact. Vari...
research
12/15/2019

Integral Mixabilty: a Tool for Efficient Online Aggregation of Functional and Probabilistic Forecasts

In this paper we extend the setting of the online prediction with expert...
research
06/10/2019

FaRM: Fair Reward Mechanism for Information Aggregation in Spontaneous Localized Settings (Extended Version)

Although peer prediction markets are widely used in crowdsourcing to agg...

Please sign up or login with your details

Forgot password? Click here to reset