[Paper Review] AutoSpearman: Automatically Mitigating Correlated Metrics for Interpreting Defect Models

Jirayus Jiarpakdee, Chakkrit Tantithamthavorn+1 more|arXiv (Cornell University)|Jun 26, 2018

Industrial Vision Systems and Defect Detection4 citations

TL;DR

This paper proposes AutoSpearman, an automated method that uses Spearman rank correlation and Variance Inflation Factor (VIF) analysis to mitigate correlated software metrics in defect prediction models. It demonstrates that commonly used feature selection techniques produce inconsistent and correlated metric subsets, while AutoSpearman achieves the highest subset consistency (69% median metric consistency across training samples) with only 1-2% performance drop in AUC, F-measure, and MCC.

ABSTRACT

The interpretation of defect models heavily relies on software metrics that are used to construct them. However, such software metrics are often correlated to defect models. Prior work often uses feature selection techniques to remove correlated metrics in order to improve the performance of defect models. Yet, the interpretation of defect models may be misleading if feature selection techniques produce subsets of inconsistent and correlated metrics. In this paper, we investigate the consistency and correlation of the subsets of metrics that are produced by nine commonly-used feature selection techniques. Through a case study of 13 publicly-available defect datasets, we find that feature selection techniques produce inconsistent subsets of metrics and do not mitigate correlated metrics, suggesting that feature selection techniques should not be used and correlation analyses must be applied when the goal is model interpretation. Since correlation analyses often involve manual selection of metrics by a domain expert, we introduce AutoSpearman, an automated metric selection approach based on correlation analyses. Our evaluation indicates that AutoSpearman yields the highest consistency of subsets of metrics among training samples and mitigates correlated metrics, while impacting model performance by 1-2%pts. Thus, to automatically mitigate correlated metrics when interpreting defect models, we recommend future studies use AutoSpearman in lieu of commonly-used feature selection techniques.

Motivation & Objective

• To investigate whether commonly used feature selection techniques produce consistent and non-correlated metric subsets in defect prediction.
• To evaluate the impact of feature selection on metric correlation and model interpretation reliability.
• To develop an automated alternative to manual correlation analysis that ensures consistent metric selection for interpretable defect models.
• To compare AutoSpearman’s performance and consistency against nine widely used feature selection techniques in defect prediction.

Proposed method

• AutoSpearman applies a two-step correlation mitigation process: first, it computes Spearman rank correlation coefficients between all metric pairs.
• Second, it uses Variance Inflation Factor (VIF) analysis to detect and remove metrics exhibiting multicollinearity.
• Metrics are iteratively removed based on the highest VIF values until all remaining metrics have VIF < 5.
• The method is applied to training samples from 13 public defect datasets, ensuring consistent metric selection across different random splits.
• Consistency is measured as the median percentage of metrics consistently selected across multiple training samples.
• Performance is evaluated using AUC, F-measure, and Matthew’s Correlation Coefficient (MCC), comparing AutoSpearman to nine feature selection techniques.

Experimental results

Research questions

• RQ1 RQ1: Do feature selection techniques consistently produce the same subset of metrics across different training samples in defect datasets?
• RQ2 RQ2: Do feature selection techniques effectively mitigate correlated metrics in defect datasets?
• RQ3 RQ3: What is the consistency of metric subsets produced by AutoSpearman across different training samples?
• RQ4 RQ4: How does AutoSpearman impact the performance of defect prediction models compared to feature selection techniques?

Key findings

• Feature selection techniques produce inconsistent metric subsets, with only 6–41% of metrics consistently selected across different training samples from the same dataset.
• Up to 100% of the metric subsets produced by feature selection techniques contain correlated metrics, indicating no effective mitigation of multicollinearity.
• AutoSpearman achieves the highest consistency among all evaluated methods, with a median of 69% of metrics consistently selected across training samples.
• AutoSpearman reduces metric correlation effectively while impacting model performance by only 1–2 percentage points in AUC, F-measure, and MCC.
• The study confirms that manual correlation analysis is necessary for reliable model interpretation, but its automation via AutoSpearman improves reproducibility and consistency.
• AutoSpearman outperforms all nine feature selection techniques in metric subset consistency, making it the recommended approach for interpretable defect modeling.