Stop Building Castles on a Swamp! The Crisis of Reproducing Automatic Search in Evidence-based Software Engineering

03/02/2021
by   Zheng Li, et al.
0

The evidence-based approach has increasingly been employed to synthesize empirical findings from the primary research in software engineering. Nevertheless, the reproducibility of evidence-based software engineering (EBSE) studies seems to be underemphasized. In our investigation into the automatic search of 311 sample studies, more than 50 reusable; about 87.5 are unrepeatable; and more than 95 implementations are unreproducible. Considering that searching is a cornerstone of an EBSE study, we are afraid that the reproducibility of the current secondary research could be worse than we can imagine. By analyzing and reporting the root causes of the aforementioned observations, we urge collaboration and cooperation among all the stakeholders in our community to improve the research reproducibility in EBSE.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/20/2023

Fingerprinting and Building Large Reproducible Datasets

Obtaining a relevant dataset is central to conducting empirical studies ...
research
11/22/2019

Four presumed gaps in the software engineering research community's knowledge

Background: The state of the art in software engineering consists of a m...
research
02/01/2023

Under the Bridge: Trolling and the Challenges of Recruiting Software Developers for Empirical Research Studies

Much of software engineering research focuses on tools, algorithms, and ...
research
08/19/2019

Reducing the Effort for Systematic Reviews in Software Engineering

Context. Systematic Reviews (SRs) are means for collecting and synthesiz...
research
03/14/2018

Bad Smells in Software Analytics Papers

CONTEXT: There has been a rapid growth in the use of data analytics to u...
research
02/24/2022

Should I Get Involved? On the Privacy Perils of Mining Software Repositories for Research Participants

Mining Software Repositories (MSRs) is an evidence-based methodology tha...

Please sign up or login with your details

Forgot password? Click here to reset