Stop Building Castles on a Swamp! The Crisis of Reproducing Automatic Search in Evidence-based Software Engineering

by   Zheng Li, et al.

The evidence-based approach has increasingly been employed to synthesize empirical findings from the primary research in software engineering. Nevertheless, the reproducibility of evidence-based software engineering (EBSE) studies seems to be underemphasized. In our investigation into the automatic search of 311 sample studies, more than 50 reusable; about 87.5 are unrepeatable; and more than 95 implementations are unreproducible. Considering that searching is a cornerstone of an EBSE study, we are afraid that the reproducibility of the current secondary research could be worse than we can imagine. By analyzing and reporting the root causes of the aforementioned observations, we urge collaboration and cooperation among all the stakeholders in our community to improve the research reproducibility in EBSE.


page 1

page 2

page 3

page 4


The Dangerous Dogmas of Software Engineering

To legitimize itself as a scientific discipline, the software engineerin...

On Gender, Ethnicity, and Culture in Empirical Software Engineering Research

This note highlights the importance of investigating diversity aspects i...

Reducing the Effort for Systematic Reviews in Software Engineering

Context. Systematic Reviews (SRs) are means for collecting and synthesiz...

How Tertiary Studies perform Quality Assessment of Secondary Studies in Software Engineering

Context: Tertiary studies are becoming increasingly popular in software ...

Beyond the Badge: Reproducibility Engineering as a Lifetime Skill

Ascertaining reproducibility of scientific experiments is receiving incr...

Bad Smells in Software Analytics Papers

CONTEXT: There has been a rapid growth in the use of data analytics to u...