University of Oulu

Wang, Y., Mäntylä, M. V., Liu, Z., & Markkula, J. (2022). Test automation maturity improves product quality—Quantitative study of open source projects using continuous integration. Journal of Systems and Software, 188, 111259.

Test automation maturity improves product quality : quantitative study of open source projects using continuous integration

Saved in:
Author: Wang, Yuqing1; Mäntylä, Mika V.1; Liu, Zihao1;
Organizations: 1M3S research unit, University of Oulu, Pentti Kaiteran katu 1, Oulu, 90014, Finland
Format: article
Version: published version
Access: open
Online Access: PDF Full Text (PDF, 1.9 MB)
Persistent link:
Language: English
Published: Elsevier, 2022
Publish Date: 2022-06-22


The popularity of continuous integration (CI) is increasing as a result of market pressure to release product features or updates frequently. The ability of CI to deliver quality at speed depends on reliable test automation. In this paper, we present an empirical study to observe the effect of test automation maturity (assessed by standard best practices in the literature) on product quality, test automation effort, and release cycle in the CI context of open source projects. We run our test automation maturity survey and got responses from 37 open source java projects. We also mined software repositories of the same projects. The main results of regression analysis reveal that, higher levels of test automation maturity are positively associated with higher product quality (p-value=0.000624) and shorter release cycle (p-value=0.01891); There is no statistically significant evidence of increased test automation effort due to higher levels of test automation maturity and product quality. Thus, we conclude that, a potential benefit of improving test automation maturity (using standard best practices) is product quality improvement and release cycle acceleration in the CI context of open source projects. We encourage future research to extend our findings by adding more datasets with different programming languages and CI tools, closed source projects, and large-scale industrial projects. Our recommendation to practitioners (in the similar CI context) is to utilize standard best practices to improve test automation maturity.

see all

Series: Journal of systems and software
ISSN: 0164-1212
ISSN-E: 1873-1228
ISSN-L: 0164-1212
Volume: 188
Article number: 111259
DOI: 10.1016/j.jss.2022.111259
Type of Publication: A1 Journal article – refereed
Field of Science: 213 Electronic, automation and communications engineering, electronics
Funding: This work is supported by TESTOMAT Project (ITEA3 ID number 16032) funded by Business Finland under Grant Decision ID 3192/31/2017, and the foundation of Tauno Tönning (project ID 20210086). This work receives open access funding provided by the University of Oulu including Oulu University Hospital .
Copyright information: © 2022 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY license (