University of Oulu

Alireza Haghighatkhah, Mika Mäntylä, Markku Oivo, Pasi Kuvaja, Test prioritization in continuous integration environments, Journal of Systems and Software, Volume 146, 2018, Pages 80-98, ISSN 0164-1212,

Test prioritization in continuous integration environments

Saved in:
Author: Haghighatkhah, Alireza1; Mäntylä, Mika1; Oivo, Markku1;
Organizations: 1Empirical Software Engineering in Software, Systems and Services (M3S), Faculty of Information Technology and Electrical Engineering (ITEE), University of Oulu, Oulu, Finland
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 4.3 MB)
Persistent link:
Language: English
Published: Elsevier, 2018
Publish Date: 2020-08-31


Two heuristics namely diversity-based (DBTP) and history-based test prioritization (HBTP) have been separately proposed in the literature. Yet, their combination has not been widely studied in continuous integration (CI) environments. The objective of this study is to catch regression faults earlier, allowing developers to integrate and verify their changes more frequently and continuously. To achieve this, we investigated six open-source projects, each of which included several builds over a large time period. Findings indicate that previous failure knowledge seems to have strong predictive power in CI environments and can be used to effectively prioritize tests. HBTP does not necessarily need to have large data, and its effectiveness improves to a certain degree with larger history interval. DBTP can be used effectively during the early stages, when no historical data is available, and also combined with HBTP to improve its effectiveness. Among the investigated techniques, we found that history-based diversity using NCD Multiset is superior in terms of effectiveness but comes with relatively higher overhead in terms of method execution time. Test prioritization in CI environments can be effectively performed with negligible investment using previous failure knowledge, and its effectiveness can be further improved by considering dissimilarities among the tests.

see all

Series: Journal of systems and software
ISSN: 0164-1212
ISSN-E: 1873-1228
ISSN-L: 0164-1212
Volume: 146
Pages: 80 - 98
DOI: 10.1016/j.jss.2018.08.061
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Copyright information: © 2018. This manuscript version is made available under the CC-BY-NC-ND 4.0 license