[Paper Review] How to Improve Deep Learning for Software Analytics (a case study with code smell detection)

Rahul Yedida, Tim Menzies|arXiv (Cornell University)|Feb 2, 2022

Software Engineering Research4 citations

TL;DR

This paper demonstrates that feedforward neural networks enhanced with fuzzy oversampling and hyperparameter optimization achieve state-of-the-art performance in code smell detection, outperforming complex deep learning models. The approach achieves over 99% AUC for 'feature envy' and 8/10 datasets for 'misplaced class', suggesting simpler models with proper preprocessing can surpass complex learners in software analytics.

ABSTRACT

To reduce technical debt and make code more maintainable, it is important to be able to warn programmers about code smells. State-of-the-art code small detectors use deep learners, without much exploration of alternatives within that technology. One promising alternative for software analytics and deep learning is GHOST (from TSE'21) that relies on a combination of hyper-parameter optimization of feedforward neural networks and a novel oversampling technique to deal with class imbalance. The prior study from TSE'21 proposing this novel "fuzzy sampling" was somewhat limited in that the method was tested on defect prediction, but nothing else. Like defect prediction, code smell detection datasets have a class imbalance (which motivated "fuzzy sampling"). Hence, in this work we test if fuzzy sampling is useful for code smell detection. The results of this paper show that we can achieve better than state-of-the-art results on code smell detection with fuzzy oversampling. For example, for "feature envy", we were able to achieve 99+\% AUC across all our datasets, and on 8/10 datasets for "misplaced class". While our specific results refer to code smell detection, they do suggest other lessons for other kinds of analytics. For example: (a) try better preprocessing before trying complex learners (b) include simpler learners as a baseline in software analytics (c) try "fuzzy sampling" as one such baseline.

Motivation & Objective

• To evaluate whether the GHOST framework—featuring fuzzy oversampling and hyperparameter optimization—improves deep learning performance in code smell detection.
• To investigate whether simpler feedforward networks can outperform complex deep learning architectures in software analytics tasks.
• To challenge the prevailing trend of using only complex deep learning models by advocating for simpler baselines and better preprocessing.
• To provide empirical evidence that class imbalance mitigation via fuzzy sampling is effective beyond defect prediction, extending to code smell detection.
• To support reproducibility by open-sourcing code and datasets for future research.

Proposed method

• The study employs feedforward neural networks as the core deep learning architecture, chosen for their simplicity and theoretical universality in function approximation.
• Fuzzy oversampling is applied to address class imbalance in code smell datasets, improving minority class representation during training.
• Hyperparameter optimization is conducted using Bayesian optimization to tune network architecture, learning rate, and batch size for optimal performance.
• Autoencoders are used for feature engineering and dimensionality reduction, enabling better input representation before classification.
• The method is evaluated across multiple public code smell datasets, with performance measured using AUC, F1, and precision-recall metrics.
• Ablation studies confirm the necessity of fuzzy oversampling and hyperparameter tuning, showing significant performance gains over baseline deep learning and traditional oversampling.

Experimental results

Research questions

• RQ1 RQ1: Can fuzzy oversampling achieve state-of-the-art results in code smell detection compared to existing deep learning methods?
• RQ2 RQ2: Why do feedforward networks with fuzzy oversampling perform so well in software analytics tasks?
• RQ3 RQ3: Is the performance gain due to the architecture, the oversampling technique, or the hyperparameter tuning?
• RQ4 RQ4: Can simpler models like feedforward networks outperform complex deep learning models when properly tuned and preprocessed?
• RQ5 RQ5: Is the GHOST framework generalizable beyond defect prediction to other software analytics tasks?

Key findings

• Fuzzy oversampling combined with hyperparameter optimization achieved over 99% AUC for the 'feature envy' code smell across all evaluated datasets.
• For the 'misplaced class' smell, the method achieved state-of-the-art performance on 8 out of 10 datasets, significantly outperforming prior approaches.
• Feedforward networks with fuzzy sampling outperformed more complex deep learning models such as RNNs and CNNs in code smell detection tasks.
• The study confirms that proper preprocessing, including fuzzy oversampling, is more impactful than architectural complexity in software analytics.
• The results suggest that simpler models with careful tuning and data augmentation can surpass complex models, challenging the assumption that deeper networks are always better.
• The GHOST framework is generalizable beyond defect prediction, as it achieves SOTA results in code smell detection, indicating broader applicability in software analytics.