Use and Misuse of the Term Experiment in Mining Software Repositories Research

10/18/2021
by   Claudia Ayala, et al.
0

The significant momentum and importance of Mining Software Repositories (MSR) in Software Engineering (SE) has fostered new opportunities and challenges for extensive empirical research. However, MSR researchers seem to struggle to characterize the empirical methods they use into the existing empirical SE body of knowledge. This is especially the case of MSR experiments. To provide evidence on the special characteristics of MSR experiments and their differences with experiments traditionally acknowledged in SE so far, we elicited the hallmarks that differentiate an experiment from other types of empirical studies and characterized the hallmarks and types of experiments in MSR. We analyzed MSR literature obtained from a small-scale systematic mapping study to assess the use of the term experiment in MSR. We found that 19 papers claiming to be an experiment are indeed not an experiment at all but also observational studies, so they use the term in a misleading way. From the remaining 81 experiment while the others stand for experiments with limited control. MSR researchers tend to overlook such limitations, compromising the interpretation of the results of their studies. We provide recommendations and insights to support the improvement of MSR experiments.

READ FULL TEXT
research
09/24/2019

Landscaping Systematic Mapping Studies in Software Engineering: A Tertiary Study

Context: A number of Systematic Mapping Studies (SMSs) that cover Softwa...
research
07/27/2020

Case Survey Studies in Software Engineering Research

Background: Given the social aspects of Software Engineering (SE), in th...
research
11/27/2019

Benefitting from the Grey Literature in Software Engineering Research

Researchers generally place the most trust in peer-reviewed, published i...
research
07/18/2018

Moving Beyond the Mean: Analyzing Variance in Software Engineering Experiments

Software Engineering (SE) experiments are traditionally analyzed with st...
research
11/15/2021

Revisiting C.S.Peirce's Experiment: 150 Years Later

An iconoclastic philosopher and polymath, Charles Sanders Peirce (1837-1...
research
09/21/2022

Unifying Classification Schemes for Software Engineering Meta-Research

Background: Classifications in meta-research enable researchers to cope ...
research
05/06/2020

Beware the Normative Fallacy

Behavioral research can provide important insights for SE practices. But...

Please sign up or login with your details

Forgot password? Click here to reset