University of Oulu

Salman, I., Turhan, B. & Vegas, S. Empir Software Eng (2018).

A controlled experiment on time pressure and confirmation bias in functional software testing

Saved in:
Author: Salman, Iflaah1; Turhan, Burak2; Vegas, Sira3
Organizations: 1M3S Group, University of Oulu
2Faculty of Information Technology, Monash University
3Escuela Tecnica Superior de Ingenieros Informaticos, Universidad Politecnica de Madrid
Format: article
Version: published version
Access: open
Online Access: PDF Full Text (PDF, 2 MB)
Persistent link:
Language: English
Published: Springer Nature, 2018
Publish Date: 2019-01-11


Context: Confirmation bias is a person’s tendency to look for evidence that strengthens his/her prior beliefs rather than refutes them. Manifestation of confirmation bias in software testing may have adverse effects on software quality. Psychology research suggests that time pressure could trigger confirmation bias.

Objective: In the software industry, this phenomenon may deteriorate software quality. In this study, we investigate whether testers manifest confirmation bias and how it is affected by time pressure in functional software testing.

Method: We performed a controlled experiment with 42 graduate students to assess manifestation of confirmation bias in terms of the conformity of their designed test cases to the provided requirements specification. We employed a one factor with two treatments between-subjects experimental design.

Results: We observed, overall, participants designed significantly more confirmatory test cases as compared to disconfirmatory ones, which is in line with previous research. However, we did not observe time pressure as an antecedent to an increased rate of confirmatory testing behaviour.

Conclusion: People tend to design confirmatory test cases regardless of time pressure. For practice, we find it necessary that testers develop self-awareness of confirmation bias and counter its potential adverse effects with a disconfirmatory attitude. We recommend further replications to investigate the effect of time pressure as a potential contributor to the manifestation of confirmation bias.

see all

Series: Empirical software engineering
ISSN: 1382-3256
ISSN-E: 1573-7616
ISSN-L: 1382-3256
Volume: 24
Issue: 4
Pages: 1727 - 1761
DOI: 10.1007/s10664-018-9668-8
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Funding: Open access funding provided by University of Oulu including Oulu University Hospital. This study was supported in part by the Infotech Oulu Doctoral grant at the University of Oulu to Iflaah Salman.
Copyright information: © The Author(s) 2018. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.