University of Oulu

Päivi Raulamo-Jurvanen, Simo Hosio, and Mika V. Mäntylä. 2019. Practitioner Evaluations on Software Testing Tools. In Proceedings of the Evaluation and Assessment on Software Engineering (EASE ’19). Association for Computing Machinery, New York, NY, USA, 57–66. DOI:

Practitioner evaluations on software testing tools

Saved in:
Author: Raulamo-Jurvanen, Päivi1; Hosio, Simo2; Mäntylä, Mika V.1
Organizations: 1M3S, University of Oulu, Oulu, Finland
2UBICOMP, University of Oulu, Oulu, Finland
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 0.6 MB)
Persistent link:
Language: English
Published: Association for Computing Machinery, 2019
Publish Date: 2020-01-14


In software engineering practice, evaluating and selecting the software testing tools that best fit the project at hand is an important and challenging task. In scientific studies of software engineering, practitioner evaluations and beliefs have recently gained interest, and some studies suggest that practitioners find beliefs of peers more credible than empirical evidence. To study how software practitioners evaluate testing tools, we applied online opinion surveys (n=89). We analyzed the reliability of the opinions utilizing Krippendorff’s alpha, intra-class correlation coefficient (ICC), and coefficients of variation (CV). Negative binomial regression was used to evaluate the effect of demographics. We find that opinions towards a specific tool can be conflicting. We show how increasing the number of respondents improves the reliability of the estimates measured with ICC. Our results indicate that on average, opinions from seven experts provide a moderate level of reliability. From demographics, we find that technical seniority leads to more negative evaluations. To improve the understanding, robustness, and impact of the findings, we need to conduct further studies by utilizing diverse sources and complementary methods.

see all

ISBN Print: 978-1-4503-7145-2
Pages: 57 - 66
DOI: 10.1145/3319008.3319018
Host publication: 23rd Evaluation and Assessment in Software Engineering Conference, EASE 2019; Copenhagen; Denmark; 14- 17 April 2019
Conference: International Conference on Evaluation and Assessment in Software Engineering
Type of Publication: A4 Article in conference proceedings
Field of Science: 113 Computer and information sciences
Funding: The work was partially supported by two research grants: No.: 3192/31/2017 for the EUREKA ITEA3 TESTOMAT project (16032) from Business Finland and No.: 286386 for the CPDSS project from the Academy of Finland.
Academy of Finland Grant Number: 286386
Detailed Information: 286386 (Academy of Finland Funding decision)
Copyright information: © 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in 23rd Evaluation and Assessment in Software Engineering Conference, EASE 2019; Copenhagen; Denmark; 14- 17 April 2019,