[Paper Review] The Effectiveness of Supervised Machine Learning Algorithms in Predicting Software Refactoring

Maurício Aniche, Erick Galani Maziero+2 more|arXiv (Cornell University)|Jan 10, 2020

Software Engineering Research4 citations

TL;DR

This study evaluates supervised machine learning algorithms for predicting software refactoring opportunities across 11,149 open-source projects. Using over two million labeled refactorings, it demonstrates that Random Forest models achieve over 90% accuracy in predicting 20 different refactoring types at class, method, and variable levels, with process and ownership metrics significantly improving model performance and generalization across diverse ecosystems.

ABSTRACT

Refactoring is the process of changing the internal structure of software to improve its quality without modifying its external behavior. Empirical studies have repeatedly shown that refactoring has a positive impact on the understandability and maintainability of software systems. However, before carrying out refactoring activities, developers need to identify refactoring opportunities. Currently, refactoring opportunity identification heavily relies on developers' expertise and intuition. In this paper, we investigate the effectiveness of machine learning algorithms in predicting software refactorings. More specifically, we train six different machine learning algorithms (i.e., Logistic Regression, Naive Bayes, Support Vector Machine, Decision Trees, Random Forest, and Neural Network) with a dataset comprising over two million refactorings from 11,149 real-world projects from the Apache, F-Droid, and GitHub ecosystems. The resulting models predict 20 different refactorings at class, method, and variable-levels with an accuracy often higher than 90%. Our results show that (i) Random Forests are the best models for predicting software refactoring, (ii) process and ownership metrics seem to play a crucial role in the creation of better models, and (iii) models generalize well in different contexts.

Motivation & Objective

• To investigate whether supervised machine learning can effectively predict software refactoring opportunities in real-world systems.
• To evaluate the performance of six ML algorithms—Logistic Regression, Naive Bayes, SVM, Decision Trees, Random Forest, and Neural Networks—in predicting 20 different refactoring operations.
• To identify which code metrics and system characteristics (especially process and ownership metrics) contribute most to model effectiveness.
• To assess the generalization capability of trained models across different software ecosystems and project types.
• To provide empirical evidence that ML-based recommendations can reduce false positives and support data-driven refactoring decisions in practice.

Proposed method

• Collected a dataset of over two million labeled refactoring operations from 11,149 real-world projects across Apache, F-Droid, and GitHub.
• Formulated refactoring prediction as a binary classification task per refactoring type (e.g., Extract Method, Rename Variable) at class, method, and variable levels.
• Extracted 142 static code metrics, including design, complexity, and ownership-related features (e.g., number of committers, file age), to represent code elements.
• Trained six supervised ML models—Logistic Regression, Naive Bayes, SVM, Decision Trees, Random Forest, and Neural Network—on the dataset using 10-fold cross-validation.
• Evaluated model performance using accuracy, precision, recall, and F1-score, with additional ablation studies to assess feature importance.
• Conducted cross-ecosystem generalization tests and time-ordered training/testing to validate model robustness and avoid data leakage.

Experimental results

Research questions

• RQ1 Which supervised machine learning algorithm performs best in predicting software refactoring operations across diverse open-source projects?
• RQ2 How do different code metrics—especially process and ownership-related features—impact the predictive performance of refactoring models?
• RQ3 To what extent can models trained on heterogeneous projects generalize to new, unseen software systems across different ecosystems?
• RQ4 How does the inclusion of temporal ordering of refactorings affect model performance and reliability?
• RQ5 What is the impact of dataset size and project diversity on model generalization and accuracy?

Key findings

• Random Forest models achieved the highest average accuracy across all 20 refactoring types, consistently outperforming other algorithms, with accuracy often exceeding 90%.
• Process and ownership metrics—such as the number of committers, file age, and change frequency—were among the most important features, significantly improving model performance.
• Models trained on data from heterogeneous ecosystems (Apache, F-Droid, GitHub) generalized well to new projects, demonstrating strong cross-project transferability.
• The time-ordered evaluation (training on first 90% of refactorings, testing on last 10%) yielded an average accuracy of 87%, confirming that temporal data leakage was not a major issue.
• Class-level refactoring models outperformed method- and variable-level models, particularly in smaller datasets (e.g., F-Droid and Apache), though the gap narrowed with larger datasets like GitHub.
• The study provides the first large-scale empirical evidence that ML models can accurately predict refactoring opportunities, offering a viable alternative to heuristic-based tools with high false positive rates.