University of Oulu

Santos, A., Vegas, S., Dieste, O. et al. A family of experiments on test-driven development. Empir Software Eng 26, 42 (2021).

A family of experiments on test-driven development

Saved in:
Author: Santos, Adrian1; Vegas, Sira2; Dieste, Oscar2;
Organizations: 1University of Oulu, Finland
2Universidad Politécnica de Madrid, Madrid, Spain
3ETAPA, Ecuador
4Istanbul Technical University, Turkey
5Blekinge Institute of Technology, Sweden
6Monash University, Australia
7University of Basilicata, Italy
8University of Bari, Italy
9University of Passau, Germany
10University of Novi Sad, Novi Sad, Serbia
11University of Tartu, Tartu, Estonia
12Ericsson, Jorvas, Finland
13Testlio, Tartu, Estonia
14ETAPA, Cuenca, Ecuador
15Paf, Helsinki, Finland
16University of Oulu, Oulu, Finland
Format: article
Version: accepted version
Access: embargoed
Persistent link:
Language: English
Published: Springer Nature, 2021
Publish Date: 2022-03-23


Context: Test-driven development (TDD) is an agile software development approach that has been widely claimed to improve software quality. However, the extent to which TDD improves quality appears to be largely dependent upon the characteristics of the study in which it is evaluated (e.g., the research method, participant type, programming environment, etc.). The particularities of each study make the aggregation of results untenable.

Objectives: The goal of this paper is to: increase the accuracy and generalizability of the results achieved in isolated experiments on TDD, provide joint conclusions on the performance of TDD across different industrial and academic settings, and assess the extent to which the characteristics of the experiments affect the quality-related performance of TDD.

Method: We conduct a family of 12 experiments on TDD in academia and industry. We aggregate their results by means of meta-analysis. We perform exploratory analyses to identify variables impacting the quality-related performance of TDD.

Results: TDD novices achieve a slightly higher code quality with iterative test-last development (i.e., ITL, the reverse approach of TDD) than with TDD. The task being developed largely determines quality. The programming environment, the order in which TDD and ITL are applied, or the learning effects from one development approach to another do not appear to affect quality. The quality-related performance of professionals using TDD drops more than for students. We hypothesize that this may be due to their being more resistant to change and potentially less motivated than students.

Conclusion: Previous studies seem to provide conflicting results on TDD performance (i.e., positive vs. negative, respectively). We hypothesize that these conflicting results may be due to different study durations, experiment participants being unfamiliar with the TDD process, or case studies comparing the performance achieved by TDD vs. the control approach (e.g., the waterfall model), each applied to develop a different system. Further experiments with TDD experts are needed to validate these hypotheses.

see all

Series: Empirical software engineering
ISSN: 1382-3256
ISSN-E: 1573-7616
ISSN-L: 1382-3256
Volume: 26
Issue: 3
Article number: 42
DOI: 10.1007/s10664-020-09895-8
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Funding: This research was developed with the support of project PGC2018-097265-B-I00, funded by: FEDER/Spanish Ministry of Science and Innovation—Research State Agency. We would like to thank the participants in the ESEIL experiments: this research would not have been possible without your help. We would also like to thank the anonymous reviewers for their valuable comments during the review of the manuscript.
Copyright information: © The Author(s), under exclusive licence to Springer Science+Business Media, LLC part of Springer Nature 2021.