University of Oulu

Seyedrebvar Hosseini, Burak Turhan, Mika Mäntylä, A benchmark study on the effectiveness of search-based data selection and feature selection for cross project defect prediction, Information and Software Technology, Volume 95, 2018, Pages 296-312, ISSN 0950-5849,

A benchmark study on the effectiveness of search-based data selection and feature selection for cross project defect prediction

Saved in:
Author: Hosseini, Seyedrebvar1; Turhan, Burak2; Mäntylä, Mika1
Organizations: 1M3S, Faculty of Information Technology and Electrical Engineering, University of Oulu, Oulu 90014, Finland
2Department of Computer Science, Brunel University London, UB8 3PH, United Kingdom
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 0.8 MB)
Persistent link:
Language: English
Published: Elsevier, 2018
Publish Date: 2020-04-27


Context: Previous studies have shown that steered training data or dataset selection can lead to better performance for cross project defect prediction(CPDP). On the other hand, feature selection and data quality are issues to consider in CPDP.

Objective: We aim at utilizing the Nearest Neighbor (NN)-Filter, embedded in genetic algorithm to produce validation sets for generating evolving training datasets to tackle CPDP while accounting for potential noise in defect labels. We also investigate the impact of using different feature sets.

Method: We extend our proposed approach, Genetic Instance Selection (GIS), by incorporating feature selection in its setting. We use 41 releases of 11 multi-version projects to assess the performance GIS in comparison with benchmark CPDP (NN-filter and Naive-CPDP) and within project (Cross-Validation(CV) and Previous Releases(PR)). To assess the impact of feature sets, we use two sets of features, SCM+OO+LOC(all) and CK+LOC(ckloc) as well as iterative info-gain subsetting(IG) for feature selection.

Results: GIS variant with info gain feature selection is significantly better than NN-Filter (all,ckloc,IG) in terms of F1 (\(\mathit{p} = \mathit{values} ≪ 0.001,\) Cohen’s \(\mathit{d} = \{0.621, 0.845, 0.762\}\)) and G (\(\mathit{p} = \mathit{values} ≪ 0.001,\) Cohen’s \(\mathit{d} = \{0.899, 1.114, 1.056\}\)), and Naive CPDP (all,ckloc,IG) in terms of F1 (\(\mathit{p} = \mathit{values} ≪ 0.001,\) Cohen’s \(\mathit{d} = \{0.743, 0.865, 0.789\}\)) and G (\(\mathit{p} = \mathit{values} ≪ 0.001,\) Cohen’s \(\mathit{d} = \{1.027, 1.119, 1.050\}\)). Overall, the performance of GIS is comparable to that of within project defect prediction (WPDP) benchmarks, i.e. CV and PR. In terms of multiple comparisons test, all variants of GIS belong to the top ranking group of approaches.

Conclusions: We conclude that datasets obtained from search based approaches combined with feature selection techniques is a promising way to tackle CPDP. Especially, the performance comparison with the within project scenario encourages further investigation of our approach. However, the performance of GIS is based on high recall in the expense of a loss in precision. Using different optimization goals, utilizing other validation datasets and other feature selection techniques are possible future directions to investigate.

see all

Series: Information and software technology
ISSN: 0950-5849
ISSN-E: 1873-6025
ISSN-L: 0950-5849
Volume: 95
Pages: 296 - 312
DOI: 10.1016/j.infsof.2017.06.004
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Copyright information: © 2017 Elsevier B.V. All rights reserved. This manuscript version is made available under the CC-BY-NC-ND 4.0 license