University of Oulu

Santos A., Oivo M., Juristo N. (2018) Moving Beyond the Mean: Analyzing Variance in Software Engineering Experiments. In: Kuhrmann M. et al. (eds) Product-Focused Software Process Improvement. PROFES 2018. Lecture Notes in Computer Science, vol 11271. Springer, Cham

Moving beyond the mean : analyzing variance in software engineering experiments

Saved in:
Author: Santos, Adrian1; Oivo, Markku1; Juristo, Natalia2
Organizations: 1M3S (M-Group), ITEE University of Oulu, Oulu, Finland
2Escuela Técnica Superior de Ingenieros Informáticos, Universidad Politécnica de Madrid, Campus Montegancedo, 28660 Boadilla del Monte, Spain
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 0.2 MB)
Persistent link:
Language: English
Published: Springer Nature, 2018
Publish Date: 2018-12-11


Software Engineering (SE) experiments are traditionally analyzed with statistical tests (e.g., t-tests, ANOVAs, etc.) that assume equally spread data across groups (i.e., the homogeneity of variances assumption). Differences across groups’ variances in SE are not seen as an opportunity to gain insights on technology performance, but instead, as a hindrance to analyze the data. We have studied the role of variance in mature experimental disciplines such as medicine. We illustrate the extent to which variance may inform on technology performance by means of simulation. We analyze a real-life industrial experiment on Test-Driven Development (TDD) where variance may impact technology desirability. Evaluating the performance of technologies just based on means—as traditionally done in SE—may be misleading. Technologies that make developers obtain similar performance (i.e., technologies with smaller variances) may be more suitable if the aim is minimizing the risk of adopting them in real practice.

see all

Series: Lecture notes in computer science
ISSN: 0302-9743
ISSN-E: 1611-3349
ISSN-L: 0302-9743
ISBN: 978-3-030-03673-7
ISBN Print: 978-3-030-03672-0
Volume: 11271
Pages: 167 - 181
DOI: 10.1007/978-3-030-03673-7_13
Host publication: Product-Focused Software Process Improvement. PROFES 2018
Conference: International Conference on Product-Focused Software Process Improvement
Type of Publication: A4 Article in conference proceedings
Field of Science: 113 Computer and information sciences
Funding: This research was developed with the support of the Spanish Ministry of Science and Innovation project TIN2014-60490-P.
Copyright information: © Springer Nature Switzerland AG 2018. This is a post-peer-review, pre-copyedit version of an article published in Lecture Notes in Computer Science, vol 11271. The final authenticated version is available online at: