Guidelines for including the grey literature and conducting multivocal literature reviews in software engineering

by   Vahid Garousi, et al.

Context: A Multivocal Literature Review (MLR) is a form of a Systematic Literature Review (SLR) which includes the grey literature (e.g., blog posts and white papers) in addition to the published (formal) literature (e.g., journal and conference papers). MLRs are useful for both researchers and practitioners since they provide summaries both the state-of-the art and -practice in a given area. Objective: There are several guidelines to conduct SLR studies in SE. However, given the facts that several phases of MLRs differ from those of traditional SLRs, for instance with respect to the search process and source quality assessment. Therefore, SLR guidelines are only partially useful for conducting MLR studies. Our goal in this paper is to present guidelines on how to conduct MLR studies in SE. Method: To develop the MLR guidelines, we benefit from three inputs: (1) existing SLR guidelines in SE, (2), a literature survey of MLR guidelines and experience papers in other fields, and (3) our own experiences in conducting several MLRs in SE. All derived guidelines are discussed in the context of three examples MLRs as running examples (two from SE and one MLR from the medical sciences). Results: The resulting guidelines cover all phases of conducting and reporting MLRs in SE from the planning phase, over conducting the review to the final reporting of the review. In particular, we believe that incorporating and adopting a vast set of recommendations from MLR guidelines and experience papers in other fields have enabled us to propose a set of guidelines with solid foundations. Conclusion: Having been developed on the basis of three types of solid experience and evidence, the provided MLR guidelines support researchers to effectively and efficiently conduct new MLRs in any area of SE.



There are no comments yet.


page 9

page 23


Guidelines for conducting multivocal literature reviews in software engineering

Context: A Multivocal Literature Review (MLR) is a form of a Systematic ...

Guidelines for the Search Strategy to Update Systematic Literature Reviews in Software Engineering

Context: Systematic Literature Reviews (SLRs) have been adopted within S...

Towards a Framework for the Design, Implementation and Reporting of Methodology Scoping Reviews

Background: In view of the growth of published papers, there is an incre...

Comparative Layouts Revisited: Design Space, Guidelines, and Future Directions

We present a systematic review on three comparative layouts (i.e., juxta...

What Evidence We Would Miss If We Do Not Use Grey Literature?

Context: Over the last years, Grey Literature (GL) is gaining increasing...

A Procedure and Guidelines for Analyzing Groups of Software Engineering Replications

Context: Researchers from different groups and institutions are collabor...

Establishing a Search String to Detect Secondary Studies in Software Engineering

Search for secondary studies is essential to establish whether the revie...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.