University of Oulu

Vahid Garousi, Michael Felderer, Mika V. Mäntylä, Guidelines for including grey literature and conducting multivocal literature reviews in software engineering, Information and Software Technology, Volume 106, 2019, Pages 101-121, ISSN 0950-5849,

Guidelines for including grey literature and conducting multivocal literature reviews in software engineering

Saved in:
Author: Garousi, Vahid1; Felderer, Michael2,3; Mäntylä, Mika V.4
Organizations: 1Information Technology Group, Wageningen University, Netherlands
2University of Innsbruck, Austria
3Blekinge Institute of Technology, Sweden
4M3S, Faculty of Information Technology and Electrical Engineering University of Oulu, Oulu, Finland
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 0.8 MB)
Persistent link:
Language: English
Published: Elsevier, 2019
Publish Date: 2020-09-20


Context: A Multivocal Literature Review (MLR) is a form of a Systematic Literature Review (SLR) which includes the grey literature (e.g., blog posts, videos and white papers) in addition to the published (formal) literature (e.g., journal and conference papers). MLRs are useful for both researchers and practitioners since they provide summaries both the state-of-the art and –practice in a given area. MLRs are popular in other fields and have recently started to appear in software engineering (SE). As more MLR studies are conducted and reported, it is important to have a set of guidelines to ensure high quality of MLR processes and their results.

Objective: There are several guidelines to conduct SLR studies in SE. However, several phases of MLRs differ from those of traditional SLRs, for instance with respect to the search process and source quality assessment. Therefore, SLR guidelines are only partially useful for conducting MLR studies. Our goal in this paper is to present guidelines on how to conduct MLR studies in SE.

Method: To develop the MLR guidelines, we benefit from several inputs: (1) existing SLR guidelines in SE, (2), a literature survey of MLR guidelines and experience papers in other fields, and (3) our own experiences in conducting several MLRs in SE. We took the popular SLR guidelines of Kitchenham and Charters as the baseline and extended/adopted them to conduct MLR studies in SE. All derived guidelines are discussed in the context of an already-published MLR in SE as the running example.

Results: The resulting guidelines cover all phases of conducting and reporting MLRs in SE from the planning phase, over conducting the review to the final reporting of the review. In particular, we believe that incorporating and adopting a vast set of experience-based recommendations from MLR guidelines and experience papers in other fields have enabled us to propose a set of guidelines with solid foundations.

Conclusion: Having been developed on the basis of several types of experience and evidence, the provided MLR guidelines will support researchers to effectively and efficiently conduct new MLRs in any area of SE. The authors recommend the researchers to utilize these guidelines in their MLR studies and then share their lessons learned and experiences.

see all

Series: Information and software technology
ISSN: 0950-5849
ISSN-E: 1873-6025
ISSN-L: 0950-5849
Volume: 106
Pages: 101 - 121
DOI: 10.1016/j.infsof.2018.09.006
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Funding: The third author has been partially supported by the Academy of Finland Grant no 298020 (Auto-Time) and by TEKES Grant no 3192/31/2017 (ITEA3: 16032 TESTOMAT project).
Academy of Finland Grant Number: 298020
Detailed Information: 298020 (Academy of Finland Funding decision)
Copyright information: © 2018 Elsevier B.V. All rights reserved. This manuscript version is made available under the CC-BY-NC-ND 4.0 license