Päivi Raulamo-Jurvanen, Simo Hosio, and Mika V. Mäntylä. 2019. Practitioner Evaluations on Software Testing Tools. In Proceedings of the Evaluation and Assessment on Software Engineering (EASE ’19). Association for Computing Machinery, New York, NY, USA, 57–66. DOI:https://doi.org/10.1145/3319008.3319018
Practitioner evaluations on software testing tools
|Author:||Raulamo-Jurvanen, Päivi1; Hosio, Simo2; Mäntylä, Mika V.1|
1M3S, University of Oulu, Oulu, Finland
2UBICOMP, University of Oulu, Oulu, Finland
|Online Access:||PDF Full Text (PDF, 0.6 MB)|
|Persistent link:|| http://urn.fi/urn:nbn:fi-fe202001142020
Association for Computing Machinery,
|Publish Date:|| 2020-01-14
In software engineering practice, evaluating and selecting the software testing tools that best fit the project at hand is an important and challenging task. In scientific studies of software engineering, practitioner evaluations and beliefs have recently gained interest, and some studies suggest that practitioners find beliefs of peers more credible than empirical evidence. To study how software practitioners evaluate testing tools, we applied online opinion surveys (n=89). We analyzed the reliability of the opinions utilizing Krippendorff’s alpha, intra-class correlation coefficient (ICC), and coefficients of variation (CV). Negative binomial regression was used to evaluate the effect of demographics. We find that opinions towards a specific tool can be conflicting. We show how increasing the number of respondents improves the reliability of the estimates measured with ICC. Our results indicate that on average, opinions from seven experts provide a moderate level of reliability. From demographics, we find that technical seniority leads to more negative evaluations. To improve the understanding, robustness, and impact of the findings, we need to conduct further studies by utilizing diverse sources and complementary methods.
|Pages:||57 - 66|
23rd Evaluation and Assessment in Software Engineering Conference, EASE 2019; Copenhagen; Denmark; 14- 17 April 2019
International Conference on Evaluation and Assessment in Software Engineering
|Type of Publication:||
A4 Article in conference proceedings
|Field of Science:||
113 Computer and information sciences
The work was partially supported by two research grants: No.: 3192/31/2017 for the EUREKA ITEA3 TESTOMAT project (16032) from Business Finland and No.: 286386 for the CPDSS project from the Academy of Finland.
|Academy of Finland Grant Number:||
286386 (Academy of Finland Funding decision)
© 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in 23rd Evaluation and Assessment in Software Engineering Conference, EASE 2019; Copenhagen; Denmark; 14- 17 April 2019, https://doi.org/10.1145/3319008.3319018.