University of Oulu

Raulamo-Jurvanen P., Hosio S., Mäntylä M.V. (2019) Applying Surveys and Interviews in Software Test Tool Evaluation. In: Franch X., Männistö T., Martínez-Fernández S. (eds) Product-Focused Software Process Improvement. PROFES 2019. Lecture Notes in Computer Science, vol 11915. Springer, Cham.

Applying surveys and interviews in software test tool evaluation

Saved in:
Author: Raulamo-Jurvanen, Päivi1; Hosio, Simo2; Mäntylä, Mika V.1
Organizations: 1ITEE, M3S, University of Oulu, Oulu, Finland
2ITEE, UBICOMP, University of Oulu, Oulu, Finland
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 0.6 MB)
Persistent link:
Language: English
Published: Springer Nature, 2019
Publish Date: 2020-11-18


Despite the multitude of available software testing tools, literature lists lack of right tools and costs as problems for adopting a tool. We conducted a case study to analyze how a group of practitioners, familiar with Robot Framework (an open source, generic test automation framework), evaluate the tool. We based the case and the unit of analysis on our academia-industry relations, i.e., availability. We used a survey (n = 68) and interviews (n = 6) with convenience sampling to develop a comprehensive view of the phenomena. The study reveals the importance of understanding the interconnection of different criteria and the potency of the context on those. Our results show that unconfirmed or unfocused opinions about criteria, e.g., about Costs or Programming Skills, can lead to misinterpretations or hamper strategic decisions if overlooking required technical competence. We conclude surveys can serve as a useful instrument for collecting empirical knowledge about tool evaluation, but experiential reasoning collected with a complementary method is required to develop into comprehensive understanding about it.

see all

Series: Lecture notes in computer science
ISSN: 0302-9743
ISSN-E: 1611-3349
ISSN-L: 0302-9743
ISBN: 978-3-030-35333-9
ISBN Print: 978-3-030-35332-2
Pages: 20 - 36
DOI: 10.1007/978-3-030-35333-9_2
Host publication: Product-Focused Software Process Improvement : 20th International Conference, PROFES 2019, Barcelona, Spain, November 27–29, 2019, Proceedings
Host publication editor: Franch, Xavier
Männistö, Tomi
Martínez-Fernández, Silverio
Conference: International Conference on Product-Focused Software Process Improvement
Type of Publication: A4 Article in conference proceedings
Field of Science: 113 Computer and information sciences
Funding: The work was supported partially by research Grants No.: 3192/31/2017 from Business Finland for the EUREKA ITEA3 TESTOMAT project (16032), and No.: 286386-CPDSS from the Academy of Finland for the CPDSS project.
Academy of Finland Grant Number: 286386
Detailed Information: 286386 (Academy of Finland Funding decision)
Copyright information: © Springer Nature Switzerland AG 2019. This is a post-peer-review, pre-copyedit version of an article published in Product-Focused Software Process Improvement : 20th International Conference, PROFES 2019, Barcelona, Spain, November 27–29, 2019, Proceedings. The final authenticated version is available online at: