University of Oulu

Wang, Y, Mäntylä, MV, Liu, Z, Markkula, J, Raulamo-jurvanen, P. Improving test automation maturity: A multivocal literature review. Softw Test Verif Reliab. 2022; 32:e1804.

Improving test automation maturity : a multivocal literature review

Saved in:
Author: Wang, Yuqing1; Mäntylä, Mika V.1; Liu, Zihao1;
Organizations: 1M3S research unit, University of Oulu, Pentti Kaiteran katu 1, Oulu, Finland
Format: article
Version: published version
Access: open
Online Access: PDF Full Text (PDF, 10.2 MB)
Persistent link:
Language: English
Published: John Wiley & Sons, 2022
Publish Date: 2022-07-19


Mature test automation is key for achieving software quality at speed. In this paper, we present a multivocal literature review with the objective to survey and synthesize the guidelines given in the literature for improving test automation maturity. We selected and reviewed 81 primary studies, consisting of 26 academic literature and 55 grey literature sources. From primary studies, we extracted 26 test automation best practices (e.g., Define an effective test automation strategy, Set up good test environments, and Develop high-quality test scripts) and collected many pieces of advice (e.g., in forms of implementation/improvement approaches, technical techniques, concepts, and experience-based heuristics) on how to conduct these best practices. We made main observations: (1) There are only six best practices whose positive effect on maturity improvement have been evaluated by academic studies using formal empirical methods; (2) several technical related best practices in this MLR were not presented in test maturity models; (3) some best practices can be linked to success factors and maturity impediments proposed by other scholars; (4) most pieces of advice on how to conduct proposed best practices were identified from experience studies and their effectiveness need to be further evaluated with cross-site empirical evidence using formal empirical methods; (5) in the literature, some advice on how to conduct certain best practices are conflicting, and some advice on how to conduct certain best practices still need further qualitative analysis.

see all

Series: Software testing, verification & reliability
ISSN: 0960-0833
ISSN-E: 1099-1689
ISSN-L: 0960-0833
Volume: 32
Issue: 3
Article number: e1804
DOI: 10.1002/stvr.1804
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Funding: This work is supported by TESTOMAT Project (ITEA3 ID number 16032) funded by Business Finland (under grant decision ID 3192/31/2017), and the foundation of Tauno Tönning (project ID 20210086).
Copyright information: © 2022 The Authors. Software Testing, Verification & Reliability published by John Wiley & Sons Ltd. This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.