University of Oulu

Y. Wang, M. Pyhäjärvi and M. V. Mäntylä, "Test Automation Process Improvement in a DevOps Team: Experience Report," 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), Porto, Portugal, 2020, pp. 314-321, doi: 10.1109/ICSTW50294.2020.00057

Test automation process improvement in a DevOps team : experience report

Saved in:
Author: Wang, Yuqing1; Pyhäjärvi, Maaret2; Mäntylä, Mika V.1
Organizations: 1M3S research unit, University of Oulu Oulu, Finland
2F-Secure Helsinki, Finland
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 0.2 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2020
Publish Date: 2021-02-19


How to successfully conduct test automation process improvement (TAPI) for continuous development, consisting of iterative software development, continuous testing, and delivery, is the challenge faced by many software organizations. In this paper, we present an experience report on TAPI in one DevOps team in F-Secure (a Finnish software company). The team builds Windows application software and exists in F-Secure’s TAPI culture. The team self-reports high satisfaction and maturity in test automation for continuous development. To study their TAPI, we reviewed a collection of experience notes, team reflection reports and telemetry result reports. Then several meetings were held to discuss the details. We found that based on the understanding of the team, test automation maturity for continuous development is defined as a set of indicators, e.g., the increasing speed to release, improving the productivity of the team, high test efficiency. Second, the team indicated that a set of critical success factors have a major impact on successfully carrying out its TAPI, e.g., incremental approach, the whole team effort, test tool choice and architecture, telemetry. Third, we compare the TAPI practices in the observed team with the practices described in prior literature. The team believes that the existing test automation maturity approaches should include the identified practices like the whole team effort to build a more comprehensive test automation improvement model for the software industry.

see all

ISBN: 978-1-7281-1075-2
ISBN Print: 978-1-7281-1076-9
Pages: 314 - 321
Article number: 9155991
DOI: 10.1109/ICSTW50294.2020.00057
Host publication: 13th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2020
Conference: IEEE International Conference on Software Testing, Verification and Validation Workshops
Type of Publication: A4 Article in conference proceedings
Field of Science: 113 Computer and information sciences
213 Electronic, automation and communications engineering, electronics
Funding: The first and third author of this study are supported by TESTOMAT Project (ITEA3 ID number 16032), funded by Business Finland under Grant Decision ID 3192/31/2017.
Copyright information: © 2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.