University of Oulu

I. Salman, P. Rodríguez, B. Turhan, A. Tosun and A. Güreller, "What Leads to a Confirmatory or Disconfirmatory Behavior of Software Testers?," in IEEE Transactions on Software Engineering, vol. 48, no. 4, pp. 1351-1368, 1 April 2022, doi: 10.1109/TSE.2020.3019892

What leads to a confirmatory or disconfirmatory behaviour of software testers?

Saved in:
Author: Salman, Iflaah1; Rodríguez, Pilar1,2; Turhan, Burak3,1;
Organizations: 1M3S Group, University of Oulu, Oulu, Finland
2Escuela Tecnica Superior de Ingenieros Informaticos, Universidad Politecnica de Madrid, Madrid, Spain
3Faculty of Information Technology, Monash University, VIC, Australia
4Faculty of Computer and Informatics Engineering, Istanbul Technical University, Istanbul, Turkey
5Ericsson, Istanbul, Turkey
Format: article
Version: published version
Access: open
Online Access: PDF Full Text (PDF, 1.6 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2022
Publish Date: 2020-09-07


Background: The existing literature in software engineering reports adverse effects of confirmation bias on software testing. Confirmation bias among software testers leads to confirmatory behaviour, which is designing or executing relatively more specification consistent test cases (confirmatory behaviour) than specification inconsistent test cases (disconfirmatory behaviour).

Objective: We aim to explore the antecedents to confirmatory and disconfirmatory behaviour of software testers. Furthermore, we aim to understand why and how those antecedents lead to (dis)confirmatory behaviour.

Method: We follow grounded theory method for the analyses of the data collected through semi-structured interviews with twelve software testers.

Results: We identified twenty antecedents to (dis)confirmatory behaviour, and classified them in nine categories. Experience and Time are the two major categories. Experience is a disconfirmatory category, which also determines which behaviour (confirmatory or disconfirmatory) occurs first among software testers, as an effect of other antecedents. Time Pressure is a confirmatory antecedent of the Time category. It also contributes to the confirmatory effects of antecedents of other categories.

Conclusion: The disconfirmatory antecedents, especially that belong to the testing process, e.g., test suite reviews by project team members, may help circumvent the deleterious effects of confirmation bias in software testing. If a team’s resources permit, the designing and execution of a test suite could be divided among the test team members, as different perspectives of testers may help to detect more errors. The results of our study are based on a single context where dedicated testing teams focus on higher levels of testing. The study’s scope does not account for the testing performed by developers. Future work includes exploring other contexts to extend our results.

see all

Series: IEEE transactions on software engineering
ISSN: 0098-5589
ISSN-E: 1939-3520
ISSN-L: 0098-5589
Volume: 48
Issue: 4
Pages: 1351 - 1368
DOI: 10.1109/TSE.2020.3019892
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Funding: The authors would like to thank the participants of the Company-ICT for this study; Alper Corlan, Basak Kahraman, Berkay Sertoglu, Cagatay Ince, Elif Deniz, Emin Vilgenoglu, Gulden Karakoyun, Ozgul Ozcan, Selda Aydin, Sezen Kaya, Sinan Verdi and Ugur Ozcan. This study was supported in part by the Infotech Oulu Doctoral grant at the University of Oulu to Iflaah Salman.
Copyright information: © 2020 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see