Can laypeople predict the replicability of social science studies without expert intervention: an exploratory study

by   Juntao Wang, et al.

The low replication rate of published studies has long concerned the social science community, making understanding the replicability a critical problem. Several studies have shown that relevant research communities can make predictions about the replicability of individual studies with above-chance accuracy. Follow-up work further indicates that laypeople can also achieve above-chance accuracy in predicting replicability when experts interpret the studies into short descriptions that are more accessible for laypeople. The involvement of scarce expert resources may make these methods expensive from financial and time perspectives. In this work, we explored whether laypeople can predict the replicability of social science studies without expert intervention. We presented laypeople with raw materials truncated from published social science papers and elicited their answers to questions related to the paper. Our results suggested that laypeople were engaged in this technical task, providing reasonable and self-contained answers. The majority of them also demonstrated a good understanding of the material. However, the solicited information had limited predictive power on the actual replication outcomes. We further discuss several lessons we learned compared to the approach with expert intervention to inspire future works.


page 1

page 2

page 3

page 4


You Can't Publish Replication Studies (and How to Anyways)

Reproducibility has been increasingly encouraged by communities of scien...

Predicting replicability – analysis of survey and prediction market data from large-scale forecasting projects

The reproducibility of published research has become an important topic ...

AI-Based Automated Speech Therapy Tools for persons with Speech Sound Disorders: A Systematic Literature Review

This paper presents a systematic literature review of published studies ...

The wisdom of the few: Predicting collective success from individual behavior

Can we predict the future success of a product, service, or business by ...

Algorithmic Social Intervention

Social and behavioral interventions are a critical tool for governments ...

Who are CUIs Really For? Representation and Accessibility in the Conversational User Interface Literature

The theme for CUI 2023 is 'designing for inclusive conversation', but wh...

Replicability of Simulation Studies for the Investigation of Statistical Methods: The RepliSims Project

Results of simulation studies evaluating the performance of statistical ...

Please sign up or login with your details

Forgot password? Click here to reset