The impact of automated feature selection techniques on the interpretation of defect models
- PDF / 5,446,041 Bytes
- 49 Pages / 439.642 x 666.49 pts Page_size
- 35 Downloads / 165 Views
The impact of automated feature selection techniques on the interpretation of defect models Jirayus Jiarpakdee1
· Chakkrit Tantithamthavorn1 · Christoph Treude2
© Springer Science+Business Media, LLC, part of Springer Nature 2020
Abstract The interpretation of defect models heavily relies on software metrics that are used to construct them. Prior work often uses feature selection techniques to remove metrics that are correlated and irrelevant in order to improve model performance. Yet, conclusions that are derived from defect models may be inconsistent if the selected metrics are inconsistent and correlated. In this paper, we systematically investigate 12 automated feature selection techniques with respect to the consistency, correlation, performance, computational cost, and the impact on the interpretation dimensions. Through an empirical investigation of 14 publicly-available defect datasets, we find that (1) 94–100% of the selected metrics are inconsistent among the studied techniques; (2) 37–90% of the selected metrics are inconsistent among training samples; (3) 0–68% of the selected metrics are inconsistent when the feature selection techniques are applied repeatedly; (4) 5–100% of the produced subsets of metrics contain highly correlated metrics; and (5) while the most important metrics are inconsistent among correlation threshold values, such inconsistent most important metrics are highly-correlated with the Spearman correlation of 0.85–1. Since we find that the subsets of metrics produced by the commonly-used feature selection techniques (except for AutoSpearman) are often inconsistent and correlated, these techniques should be avoided when interpreting defect models. In addition to introducing AutoSpearman which mitigates correlated metrics better than commonly-used feature selection techniques, this paper opens up new research avenues in the automated selection of features for defect models to optimise for interpretability as well as performance. Keywords Software analytics · Defect prediction · Model interpretation · Feature selection Communicated by: Tim Menzies Jirayus Jiarpakdee
[email protected] Chakkrit Tantithamthavorn [email protected] Christoph Treude [email protected] 1
Faculty of Information and Technology, Monash University, Melbourne, Australia
2
School of Computer Science, The University of Adelaide, Adelaide, Australia
Empirical Software Engineering
1 Introduction Defect models are statistical or machine learning models that are used to investigate the impact of software metrics (e.g., lines of code) on defect-proneness and to identify defectprone software modules. The interpretation of defect models is used to validate hypotheses to develop empirical theories related to software quality, which are essential to chart quality improvement plans (Jiarpakdee et al. 2018a, 2020). The conclusions of defect models heavily rely on the studied software metrics. However, software metrics often have strong correlation among themselves (Gil and Lalouche 2017; J
Data Loading...
