DeepAI AI Chat
Log In Sign Up

Predicting the Reproducibility of Social and Behavioral Science Papers Using Supervised Learning Models

by   Jian Wu, et al.
Old Dominion University
Penn State University
Texas A&M University

In recent years, significant effort has been invested verifying the reproducibility and robustness of research claims in social and behavioral sciences (SBS), much of which has involved resource-intensive replication projects. In this paper, we investigate prediction of the reproducibility of SBS papers using machine learning methods based on a set of features. We propose a framework that extracts five types of features from scholarly work that can be used to support assessments of reproducibility of published research claims. Bibliometric features, venue features, and author features are collected from public APIs or extracted using open source machine learning libraries with customized parsers. Statistical features, such as p-values, are extracted by recognizing patterns in the body text. Semantic features, such as funding information, are obtained from public APIs or are extracted using natural language processing models. We analyze pairwise correlations between individual features and their importance for predicting a set of human-assessed ground truth labels. In doing so, we identify a subset of 9 top features that play relatively more important roles in predicting the reproducibility of SBS papers in our corpus. Results are verified by comparing performances of 10 supervised predictive classifiers trained on different sets of features.


page 4

page 11

page 12

page 14


ReproServer: Making Reproducibility Easier and Less Intensive

Reproducibility in the computational sciences has been stymied because o...

Reproducibility Signals in Science: A preliminary analysis

Reproducibility is an important feature of science; experiments are rete...

Leakage and the Reproducibility Crisis in ML-based Science

The use of machine learning (ML) methods for prediction and forecasting ...

Reproducibility in Machine Learning for Health

Machine learning algorithms designed to characterize, monitor, and inter...

A Synthetic Prediction Market for Estimating Confidence in Published Work

Explainably estimating confidence in published scholarly work offers opp...

A Machine Learning Framework for Automatic Prediction of Human Semen Motility

In this paper, human semen samples from the visem dataset collected by t...