Predicting replicability – analysis of survey and prediction market data from large-scale forecasting projects

by   Michael Gordon, et al.

The reproducibility of published research has become an important topic in science policy. A number of large-scale replication projects have been conducted to gauge the overall reproducibility in specific academic fields. Here, we present an analysis of data from four studies which sought to forecast the outcomes of replication projects in the social and behavioural sciences, using human experts who participated in prediction markets and answered surveys. Because the number of findings replicated and predicted in each individual study was small, pooling the data offers an opportunity to evaluate hypotheses regarding the performance of prediction markets and surveys at a higher power. In total, peer beliefs were elicited for the replication outcomes of 103 published findings. We find there is information within the scientific community about the replicability of scientific findings, and that both surveys and prediction markets can be used to elicit and aggregate this information. Our results show prediction markets can determine the outcomes of direct replications with 73 the average survey responses are correlated with outcomes (0.581 and 0.564 respectively, both p < .001). We also found a significant relationship between p-values of the original findings and replication outcomes. The dataset is made available through the R package pooledmaRket and can be used to further study community beliefs towards replications outcomes as elicited in the surveys and prediction markets.


page 1

page 2

page 3

page 4


A Synthetic Prediction Market for Estimating Confidence in Published Work

Explainably estimating confidence in published scholarly work offers opp...

Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication

The last decade saw the emergence of systematic large-scale replication ...

Can laypeople predict the replicability of social science studies without expert intervention: an exploratory study

The low replication rate of published studies has long concerned the soc...

Bayesian sample size determination for multi-site replication studies

An ongoing "reproducibility crisis" calls into question scientific disco...

You Can't Publish Replication Studies (and How to Anyways)

Reproducibility has been increasingly encouraged by communities of scien...

In-class Data Analysis Replications: Teaching Students while Testing Science

Science is facing a reproducibility crisis. Previous work has proposed i...

Arukikata Travelogue Dataset

We have constructed Arukikata Travelogue Dataset and released it free of...

Please sign up or login with your details

Forgot password? Click here to reset