University of Oulu

Haghighatkhah A., Mäntylä M., Oivo M., Kuvaja P. (2018) Test Case Prioritization Using Test Similarities. In: Kuhrmann M. et al. (eds) Product-Focused Software Process Improvement. PROFES 2018. Lecture Notes in Computer Science, vol 11271. Springer, Cham.

Test case prioritization using test similarities

Saved in:
Author: Haghighatkhah, Alireza1; Mäntylä, Mika1; Oivo, Markku1;
Organizations: 1M3S Research Unit, University of Oulu, P.O. Box 3000, 90014 Oulu, Finland
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 1 MB)
Persistent link:
Language: English
Published: Springer Nature, 2018
Publish Date: 2019-11-03


A classical heuristic in software testing is to reward diversity, which implies that a higher priority must be assigned to test cases that differ the most from those already prioritized. This approach is commonly known as similarity-based test prioritization (SBTP) and can be realized using a variety of techniques. The objective of our study is to investigate whether SBTP is more effective at finding defects than random permutation, as well as determine which SBTP implementations lead to better results. To achieve our objective, we implemented five different techniques from the literature and conducted an experiment using the defects4j dataset, which contains 395 real faults from six real-world open-source Java programs. Findings indicate that running the most dissimilar test cases early in the process is largely more effective than random permutation (Vargha–Delaney A [VDA]: 0.76–0.99 observed using normalized compression distance). No technique was found to be superior with respect to the effectiveness. Locality-sensitive hashing was, to a small extent, less effective than other SBTP techniques (VDA: 0.38 observed in comparison to normalized compression distance), but its speed largely outperformed the other techniques (i.e., it was approximately 5–111 times faster). Our results bring to mind the well-known adage, “don’t put all your eggs in one basket”. To effectively consume a limited testing budget, one should spread it evenly across different parts of the system by running the most dissimilar test cases early in the testing process.

see all

Series: Lecture notes in computer science
ISSN: 0302-9743
ISSN-E: 1611-3349
ISSN-L: 0302-9743
Pages: 243 - 259
DOI: 10.1007/978-3-030-03673-7_18
Host publication: Product-Focused Software Process Improvement. PROFES 2018
Conference: International Conference on Product-Focused Software Process Improvement
Type of Publication: A4 Article in conference proceedings
Field of Science: 113 Computer and information sciences
Copyright information: © Springer Nature Switzerland AG 2018. This is a post-peer-review, pre-copyedit version of an article published in Product-Focused Software Process Improvement. PROFES 2018. The final authenticated version is available online at: