[Paper Review] BiLO-CPDP: Bi-Level Programming for Automated Model Discovery in Cross-Project Defect Prediction

Ke Li, Zilin Xiang+2 more|arXiv (Cornell University)|Aug 31, 2020

Software Engineering Research86 references4 citations

TL;DR

This paper proposes BiLO-CPDP, the first bi-level programming framework for automated cross-project defect prediction (CPDP) model discovery. It jointly optimizes classifier and transfer learner selection with their hyper-parameters via hierarchical upper- and lower-level optimization, achieving superior performance over 21 existing CPDP techniques and Auto-Sklearn across 20 projects, especially when allocating more budget to the upper-level for model combination search.

ABSTRACT

Cross-Project Defect Prediction (CPDP), which borrows data from similar projects by combining a transfer learner with a classifier, have emerged as a promising way to predict software defects when the available data about the target project is insufficient. How-ever, developing such a model is challenge because it is difficult to determine the right combination of transfer learner and classifier along with their optimal hyper-parameter settings. In this paper, we propose a tool, dubbedBiLO-CPDP, which is the first of its kind to formulate the automated CPDP model discovery from the perspective of bi-level programming. In particular, the bi-level programming proceeds the optimization with two nested levels in a hierarchical manner. Specifically, the upper-level optimization routine is designed to search for the right combination of transfer learner and classifier while the nested lower-level optimization routine aims to optimize the corresponding hyper-parameter settings.To evaluateBiLO-CPDP, we conduct experiments on 20 projects to compare it with a total of 21 existing CPDP techniques, along with its single-level optimization variant and Auto-Sklearn, a state-of-the-art automated machine learning tool. Empirical results show that BiLO-CPDP champions better prediction performance than all other 21 existing CPDP techniques on 70% of the projects, while be-ing overwhelmingly superior to Auto-Sklearn and its single-level optimization variant on all cases. Furthermore, the unique bi-level formalization inBiLO-CPDP also permits to allocate more budget to the upper-level, which significantly boosts the performance.

Motivation & Objective

• To address the challenge of selecting optimal combinations of transfer learners and classifiers with their hyper-parameters in cross-project defect prediction (CPDP), which is often done ad-hoc in prior work.
• To overcome the limitations of existing CPDP techniques that focus only on classifier hyper-parameters or use fixed combinations, leading to suboptimal performance.
• To enable automated, systematic discovery of high-performing CPDP models through a unified optimization framework.
• To explore the impact of computational budget allocation between model architecture selection (upper-level) and hyper-parameter tuning (lower-level) in CPDP.
• To demonstrate that optimizing the combination of transfer learner and classifier is more critical than hyper-parameter tuning alone.

Proposed method

• Formulates CPDP model discovery as a bi-level optimization problem: the upper-level selects the optimal combination of transfer learner and classifier, while the lower-level tunes their respective hyper-parameters.
• Employs a hierarchical optimization structure where the lower-level optimization is nested within the upper-level, ensuring that hyper-parameters are optimized for each candidate model combination.
• Uses a configurable portfolio of transfer learners and classifiers as input, enabling systematic exploration of diverse model architectures.
• Applies a flexible budget allocation strategy, allowing more computational resources to be assigned to the upper-level to explore a broader range of model combinations.
• Employs standard machine learning evaluation metrics (AUC) and statistical tests (Scott-Knott, Wilcoxon signed-rank, effect size) to ensure robust performance comparison.
• Integrates with existing AutoML principles but extends them by explicitly modeling the interaction between model architecture and hyper-parameter optimization in CPDP.

Experimental results

Research questions

• RQ1 Can bi-level programming effectively automate the discovery of high-performing CPDP models by jointly optimizing model architecture and hyper-parameters?
• RQ2 How does the performance of BiLO-CPDP compare to 21 state-of-the-art CPDP techniques that use hand-crafted combinations and reported hyper-parameters?
• RQ3 Does BiLO-CPDP outperform Auto-Sklearn, a leading AutoML tool, in CPDP tasks?
• RQ4 What is the impact of allocating more computational budget to the upper-level optimization (model combination search) versus the lower-level (hyper-parameter tuning)?
• RQ5 Is the selection of the optimal transfer learner and classifier combination more important than hyper-parameter tuning alone in CPDP?

Key findings

• BiLO-CPDP outperforms all 21 existing CPDP techniques on 70% of the 20 evaluated projects, demonstrating its superiority in automated model discovery.
• BiLO-CPDP significantly outperforms Auto-Sklearn and its single-level optimization variant on all 20 projects, highlighting the advantage of its bi-level structure.
• Allocating more computational budget to the upper-level optimization leads to a substantial performance boost, confirming that model combination search is more impactful than hyper-parameter tuning alone.
• The empirical study shows that the optimal combination of transfer learner and classifier is more decisive than hyper-parameter settings, validating the need for joint optimization.
• The use of bi-level programming enables a more flexible and effective budget allocation strategy, allowing practitioners to prioritize model architecture exploration when needed.
• Statistical tests confirm that the performance gains of BiLO-CPDP are not due to random variation, with large effect sizes and high confidence in the results.