University of Oulu

Seyedrebvar Hosseini and Burak Turhan. 2021. A comparison of similarity based instance selection methods for cross project defect prediction. In Proceedings of the 36th Annual ACM Symposium on Applied Computing (SAC '21). Association for Computing Machinery, New York, NY, USA, 1455–1464. DOI:https://doi.org/10.1145/3412841.3442020

A comparison of similarity based instance selection methods for cross project defect prediction

Saved in:
Author: Hosseini, Seyedrebvar1; Turhan, Burak1
Organizations: 1M3S Research Unit, Faculty of ITEE, University of Oulu, Oulu, Finland
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 0.5 MB)
Persistent link: http://urn.fi/urn:nbn:fi-fe2021051229661
Language: English
Published: Association for Computing Machinery, 2021
Publish Date: 2021-05-12
Description:

Abstract

Context: Previous studies have shown that training data instance selection based on nearest neighborhood (NN) information can lead to better performance in cross project defect prediction (CPDP) by reducing heterogeneity in training datasets. However, neighborhood calculation is computationally expensive and approximate methods such as Locality Sensitive Hashing (LSH) can be as effective as exact methods.

Aim: We aim at comparing instance selection methods for CPDP, namely LSH, NN-filter, and Genetic Instance Selection (GIS).

Method: We conduct experiments with five base learners, optimizing their hyper parameters, on 13 datasets from PROMISE repository in order to compare the performance of LSH with benchmark instance selection methods NN-Filter and GIS.

Results: The statistical tests show six distinct groups for F-measure performance. The top two group contains only LSH and GIS benchmarks whereas the bottom two groups contain only NN-Filter variants. LSH and GIS favor recall more than precision. In fact, for precision performance only three significantly distinct groups are detected by the tests where the top group is comprised of NN-Filter variants only. Recall wise, 16 different groups are identified where the top three groups contain only LSH methods, four of the next six are GIS only and the bottom five contain only NN-Filter. Finally, NN-Filter benchmarks never outperform the LSH counterparts with the same base learner, tuned or non-tuned. Further, they never even belong to the same rank group, meaning that LSH is always significantly better than NN-Filter with the same learner and settings.

Conclusions: The increase in performance and the decrease in computational overhead and runtime make LSH a promising approach. However, the performance of LSH is based on high recall and in environments where precision is considered more important NN-Filter should be considered.

see all

ISBN Print: 978-1-4503-8104-8
Pages: 1455 - 1464
DOI: 10.1145/3412841.3442020
OADOI: https://oadoi.org/10.1145/3412841.3442020
Host publication: Proceedings of the 36th Annual ACM Symposium on Applied Computing
Conference: Annual ACM Symposium on Applied Computing
Type of Publication: A4 Article in conference proceedings
Field of Science: 113 Computer and information sciences
Subjects:
Copyright information: © 2021 Association for Computing Machinery. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in Proceedings of the 36th Annual ACM Symposium on Applied Computing, https://doi.org/10.1145/3412841.3442020.