[Paper Review] A Comparative Study of Different Source Code Metrics and Machine Learning Algorithms for Predicting Change Proneness of Object Oriented Systems

Lov Kumar, Ashish Sureka|arXiv (Cornell University)|Dec 21, 2017

Software Engineering Research5 references3 citations

TL;DR

This study evaluates 21 source code metrics and 11 feature selection techniques to predict change-proneness in object-oriented systems using 18 classifiers and 3 ensemble methods. The LSSVM-RBF model with metrics selected via the proposed feature selection framework (PFST) achieved the highest accuracy, outperforming other classifiers and feature selection approaches.

ABSTRACT

Change-prone classes or modules are defined as software components in the source code which are likely to change in the future. Change-proneness prediction is useful to the maintenance team as they can optimize and focus their testing resources on the modules which have a higher likelihood of change. Change-proneness prediction model can be built by using source code metrics as predictors or features within a machine learning classification framework. In this paper, twenty one source code metrics are computed to develop a statistical model for predicting change-proneness modules. Since the performance of the change-proneness model depends on the source code metrics, they are used as independent variables or predictors for the change-proneness model. Eleven different feature selection techniques (including the usage of all the $21$ proposed source code metrics described in the paper) are used to remove irrelevant features and select the best set of features. The effectiveness of the set of source code metrics are evaluated using eighteen different classiffication techniques and three ensemble techniques. Experimental results demonstrate that the model based on selected set of source code metrics after applying feature selection techniques achieves better results as compared to the model using all source code metrics as predictors. Our experimental results reveal that the predictive model developed using LSSVM-RBF yields better result as compared to other classification techniques

Motivation & Objective

• To investigate the effectiveness of 21 source code metrics in predicting change-proneness in object-oriented software.
• To evaluate the impact of eleven feature selection techniques on improving prediction model performance.
• To compare the performance of 18 machine learning classifiers and 3 ensemble methods in predicting change-prone classes.
• To identify the optimal combination of metrics, feature selection, and classification techniques for accurate change-proneness prediction.
• To provide a reproducible framework for change-proneness prediction using open-source datasets and established tools.

Proposed method

• Computed 21 object-oriented software metrics (e.g., cyclomatic complexity, coupling, depth of inheritance) using the Understand tool on Eclipse 2.0 and 2.1 plug-ins.
• Identified changed classes between versions using the Jar Compare tool to label training data as change-prone or not.
• Applied 11 feature selection techniques—including filter, wrapper, and embedded methods—to reduce irrelevant metrics and select optimal subsets.
• Trained 18 individual classifiers (e.g., SVM, Random Forest, RBFN) and 3 ensemble methods (e.g., NDTF, bagging, boosting) on selected metric subsets.
• Used Wilcoxon signed-rank tests and mean difference analysis to statistically compare model performance across techniques.
• Evaluated models using standard classification metrics such as accuracy, F1-score, and misclassification error.

Experimental results

Research questions

• RQ1 Which source code metrics significantly correlate with class change-proneness across different Eclipse plug-ins?
• RQ2 Does using a selected subset of metrics through feature selection improve change-proneness prediction accuracy compared to using all 21 metrics?
• RQ3 Which machine learning classifier yields the highest performance in predicting change-proneness after feature selection?
• RQ4 How do different feature selection techniques compare in their ability to enhance model performance?
• RQ5 How do ensemble methods compare to individual classifiers in predicting change-proneness?

Key findings

• The LSSVM-RBF classifier achieved the highest accuracy and lowest misclassification error among all 18 individual classifiers tested.
• The proposed feature selection framework (PFST) consistently outperformed other feature selection techniques in terms of mean difference in performance metrics.
• The Nonlinear Ensemble Decision Tree Forest (NDTF) ensemble method outperformed all other ensemble methods and was second only to LSSVM-RBF in overall performance.
• A small subset of source code metrics—selected via PFST—achieved higher prediction accuracy than models using all 21 metrics.
• The performance of feature selection techniques varied significantly depending on the choice of classifier, indicating strong interaction effects between feature selection and learning algorithm.
• Statistical analysis confirmed that the differences in model performance were significant, with LSSVM-RBF and PFST-based models showing the most robust results across all datasets.