What Evidence We Would Miss If We Do Not Use Grey Literature?

by   Fernando Kamei, et al.
Universidade Federal de Alagoas

Context: Over the last years, Grey Literature (GL) is gaining increasing attention in Secondary Studies in Software Engineering (SE). Notably, Multivocal Literature Review (MLR) studies, that search for evidence in both Traditional Literature (TL) and GL, is particularly benefiting from this raise of GL content. Despite the growing interest in MLR-based studies, the literature assessing how GL has contributed to MLR studies is still scarce. Objective: This research aims to assess how the use of GL contributed to MLR studies. By contributing, we mean, understanding to what extent GL is providing evidence that is indeed used by an MLR to answer its research question. Method: We conducted a tertiary study to identify MLR studies published between 2017 and 2019, selecting nine MLRs studies. Using qualitative and quantitative analysis, we identified the GL used and assessed to what extent these MLRs are contributing to MLR studies. Results: Our analysis identified that 1) GL provided evidence not found in TL, 2) most of the GL sources were used to provide recommendations to solve problems, explain a topic, and classify the findings, and 3) 19 different GL types were used in the studies; these GLs were mainly produced by SE practitioners (including blog posts, slides presentations, or project descriptions). Conclusions: We evidence how GL contributed to MLR studies. We observed that if these GLs were not included in the MLR, several findings would have been omitted or weakened. We also described the challenges involved when conducting this investigation, along with potential ways to deal with them, which may help future SE researchers.


page 1

page 2

page 3

page 4


Grey Literature in Software Engineering: A Critical Review

Context: Grey Literature (GL) recently has grown in Software Engineering...

Beware the Normative Fallacy

Behavioral research can provide important insights for SE practices. But...

Tainted Love: A Systematic Review of Online Romance Fraud

Romance fraud involves cybercriminals engineering a romantic relationshi...

Secondary Studies in the Academic Context: A Systematic Mapping and Survey

Context: Several researchers have reported their experiences in applying...

Guidelines for conducting multivocal literature reviews in software engineering

Context: A Multivocal Literature Review (MLR) is a form of a Systematic ...

Hazards in Deep Learning Testing: Prevalence, Impact and Recommendations

Much research on Machine Learning testing relies on empirical studies th...

Please sign up or login with your details

Forgot password? Click here to reset