Guidelines for conducting multivocal literature reviews in software engineering

by   Vahid Garousi, et al.

Context: A Multivocal Literature Review (MLR) is a form of a Systematic Literature Review (SLR) which includes the grey literature (e.g., blog posts and white papers) in addition to the published (formal) literature (e.g., journal and conference papers). MLRs are useful for both researchers and practitioners since they provide summaries both the state-of-the art and -practice in a given area. Objective: There are several guidelines to conduct SLR studies in SE. However, given the facts that several phases of MLRs differ from those of traditional SLRs, for instance with respect to the search process and source quality assessment. Therefore, SLR guidelines are only partially useful for conducting MLR studies. Our goal in this paper is to present guidelines on how to conduct MLR studies in SE. Method: To develop the MLR guidelines, we benefit from three inputs: (1) existing SLR guidelines in SE, (2), a literature survey of MLR guidelines and experience papers in other fields, and (3) our own experiences in conducting several MLRs in SE. All derived guidelines are discussed in the context of three examples MLRs as running examples (two from SE and one MLR from the medical sciences). Results: The resulting guidelines cover all phases of conducting and reporting MLRs in SE from the planning phase, over conducting the review to the final reporting of the review. In particular, we believe that incorporating and adopting a vast set of recommendations from MLR guidelines and experience papers in other fields have enabled us to propose a set of guidelines with solid foundations. Conclusion: Having been developed on the basis of three types of solid experience and evidence, the provided MLR guidelines support researchers to effectively and efficiently conduct new MLRs in any area of SE.


page 4

page 5

page 21

page 23


Guidelines for including the grey literature and conducting multivocal literature reviews in software engineering

Context: A Multivocal Literature Review (MLR) is a form of a Systematic ...

Towards a Framework for the Design, Implementation and Reporting of Methodology Scoping Reviews

Background: In view of the growth of published papers, there is an incre...

Guidelines for the Search Strategy to Update Systematic Literature Reviews in Software Engineering

Context: Systematic Literature Reviews (SLRs) have been adopted within S...

Inclusion and Exclusion Criteria in Software Engineering Tertiary Studies: A Systematic Mapping and Emerging Framework

Context: Tertiary studies in software engineering (TS@SE) are widely use...

What Evidence We Would Miss If We Do Not Use Grey Literature?

Context: Over the last years, Grey Literature (GL) is gaining increasing...

Retrieving and mining professional experience of software practice from grey literature: an exploratory review

Background: Retrieving and mining practitioners' self–reports of their p...

A review of heath economic evaluation practice in the Netherlands: are we moving forward?

In 2016, the Dutch National Health Care Institute issued new guidelines ...

Please sign up or login with your details

Forgot password? Click here to reset