Log In Sign Up

Guidelines for including the grey literature and conducting multivocal literature reviews in software engineering

by   Vahid Garousi, et al.

Context: A Multivocal Literature Review (MLR) is a form of a Systematic Literature Review (SLR) which includes the grey literature (e.g., blog posts and white papers) in addition to the published (formal) literature (e.g., journal and conference papers). MLRs are useful for both researchers and practitioners since they provide summaries both the state-of-the art and -practice in a given area. Objective: There are several guidelines to conduct SLR studies in SE. However, given the facts that several phases of MLRs differ from those of traditional SLRs, for instance with respect to the search process and source quality assessment. Therefore, SLR guidelines are only partially useful for conducting MLR studies. Our goal in this paper is to present guidelines on how to conduct MLR studies in SE. Method: To develop the MLR guidelines, we benefit from three inputs: (1) existing SLR guidelines in SE, (2), a literature survey of MLR guidelines and experience papers in other fields, and (3) our own experiences in conducting several MLRs in SE. All derived guidelines are discussed in the context of three examples MLRs as running examples (two from SE and one MLR from the medical sciences). Results: The resulting guidelines cover all phases of conducting and reporting MLRs in SE from the planning phase, over conducting the review to the final reporting of the review. In particular, we believe that incorporating and adopting a vast set of recommendations from MLR guidelines and experience papers in other fields have enabled us to propose a set of guidelines with solid foundations. Conclusion: Having been developed on the basis of three types of solid experience and evidence, the provided MLR guidelines support researchers to effectively and efficiently conduct new MLRs in any area of SE.


page 9

page 23


Guidelines for conducting multivocal literature reviews in software engineering

Context: A Multivocal Literature Review (MLR) is a form of a Systematic ...

Guidelines for the Search Strategy to Update Systematic Literature Reviews in Software Engineering

Context: Systematic Literature Reviews (SLRs) have been adopted within S...

Benefitting from the Grey Literature in Software Engineering Research

Researchers generally place the most trust in peer-reviewed, published i...

Towards a Framework for the Design, Implementation and Reporting of Methodology Scoping Reviews

Background: In view of the growth of published papers, there is an incre...

What Evidence We Would Miss If We Do Not Use Grey Literature?

Context: Over the last years, Grey Literature (GL) is gaining increasing...

Navigating the reporting guideline environment for computational pathology: A review

The application of new artificial intelligence (AI) discoveries is trans...