[Paper Review] Bridging Expert Knowledge with Deep Learning Techniques for Just-In-Time Defect Prediction

Xin Zhou, DongGyun Han+1 more|arXiv (Cornell University)|Mar 17, 2024

Industrial Vision Systems and Defect Detection4 citations

TL;DR

This paper proposes SimCom++, a novel fusion framework that combines expert-knowledge-based hand-crafted features (simple model) with deep learning–extracted semantic features (complex model) via early and late fusion strategies. The approach significantly outperforms state-of-the-art baselines by 5.7–26.9% in key metrics, demonstrating that integrating expert knowledge and deep learning features enhances just-in-time defect prediction performance.

ABSTRACT

Just-In-Time (JIT) defect prediction aims to automatically predict whether a commit is defective or not, and has been widely studied in recent years. In general, most studies can be classified into two categories: 1) simple models using traditional machine learning classifiers with hand-crafted features, and 2) complex models using deep learning techniques to automatically extract features from commit contents. Hand-crafted features used by simple models are based on expert knowledge but may not fully represent the semantic meaning of the commits. On the other hand, deep learning-based features used by complex models represent the semantic meaning of commits but may not reflect useful expert knowledge. Simple models and complex models seem complementary to each other to some extent. To utilize the advantages of both simple and complex models, we propose a model fusion framework that adopts both early fusions on the feature level and late fusions on the decision level. We propose SimCom++ by adopting the best early and late fusion strategies. The experimental results show that SimCom++ can significantly outperform the baselines by 5.7--26.9\%. In addition, our experimental results confirm that the simple model and complex model are complementary to each other.

Motivation & Objective

• To address the limitations of existing JIT defect prediction models that either rely solely on hand-crafted features (lacking semantic depth) or on deep learning (lacking interpretability and expert insight).
• To investigate whether combining simple models (expert knowledge) and complex models (deep learning) can yield better performance than using either approach in isolation.
• To design a multi-modal fusion framework that leverages both feature-level (early) and decision-level (late) fusion to integrate complementary strengths of simple and complex models.
• To evaluate the robustness and generalization of the proposed framework across different software projects and under varying data conditions, including exclusion of large commits.

Proposed method

• The framework employs a simple model based on hand-crafted, expert-knowledge-driven features such as code change size, churn, and historical defect rates.
• It uses a deep learning model (Com model in SimCom++) to automatically extract semantic representations from commit logs and code changes using neural networks.
• Early fusion combines raw features from both models before classification, enabling joint learning of low-level and high-level patterns.
• Late fusion combines the prediction scores from the simple and complex models using weighted averaging or stacking, allowing the model to learn optimal decision boundaries.
• The final SimCom++ model is trained end-to-end using a hybrid training strategy that optimizes both early and late fusion components.
• The framework is evaluated on multiple open-source projects using standard metrics: AUC-ROC, AUC-PR, and F1-score.

Experimental results

Research questions

• RQ1 Can combining expert-knowledge-based features with deep learning–extracted semantic features improve JIT defect prediction performance beyond state-of-the-art models?
• RQ2 Is there a complementary relationship between simple models (based on expert knowledge) and complex models (based on deep learning) in JIT defect prediction?
• RQ3 Does the proposed fusion framework maintain superior performance when large commits are excluded from the training and testing data?
• RQ4 How do early fusion and late fusion strategies compare in enhancing the predictive power of the combined model?

Key findings

• SimCom++ significantly outperforms all baseline models, achieving an average improvement of 5.7% in AUC-ROC, 12.5% in AUC-PR, and 17.9% in F1-score.
• The model improves performance by up to 26.9% in AUC-ROC compared to the strongest baseline, demonstrating the effectiveness of the fusion strategy.
• The framework consistently outperforms both the simple model and complex model alone across all evaluation metrics and projects, with average gains of 4.4%–10.1%.
• Even when large commits (top 10%–40%) are excluded, SimCom++ still outperforms the best baseline by 9.5% in AUC-ROC, 16.1% in AUC-PR, and 13.5% in F1-score, indicating robustness.
• The results confirm that expert knowledge and deep learning features are complementary, and their integration via multi-modal fusion yields superior predictive performance.
• The ablation study confirms that both early and late fusion contribute meaningfully to the final performance, with late fusion playing a particularly strong role in decision calibration.