[Paper Review] Comparative Study of Machine Learning Test Case Prioritization for Continuous Integration Testing

Dusica Marijan|arXiv (Cornell University)|Apr 22, 2022

Software Testing and Debugging Techniques4 citations

TL;DR

This study evaluates machine learning models—LambdaRank, SVM, GBDT, and ANN—for test case prioritization in continuous integration (CI), using real industrial datasets. It finds that model performance varies significantly with test history length (optimal: 40–60%) and time budget, with LambdaRank achieving the best fault-detection effectiveness under short time constraints, while ANN is most sensitive to history length but fastest in ranking time.

ABSTRACT

There is a growing body of research indicating the potential of machine learning to tackle complex software testing challenges. One such challenge pertains to continuous integration testing, which is highly time-constrained, and generates a large amount of data coming from iterative code commits and test runs. In such a setting, we can use plentiful test data for training machine learning predictors to identify test cases able to speed up the detection of regression bugs introduced during code integration. However, different machine learning models can have different fault prediction performance depending on the context and the parameters of continuous integration testing, for example variable time budget available for continuous integration cycles, or the size of test execution history used for learning to prioritize failing test cases. Existing studies on test case prioritization rarely study both of these factors, which are essential for the continuous integration practice. In this study we perform a comprehensive comparison of the fault prediction performance of machine learning approaches that have shown the best performance on test case prioritization tasks in the literature. We evaluate the accuracy of the classifiers in predicting fault-detecting tests for different values of the continuous integration time budget and with different length of test history used for training the classifiers. In evaluation, we use real-world industrial datasets from a continuous integration practice. The results show that different machine learning models have different performance for different size of test history used for model training and for different time budget available for test case execution. Our results imply that machine learning approaches for test prioritization in continuous integration testing should be carefully configured to achieve optimal performance.

Motivation & Objective

• To evaluate the fault-prediction effectiveness and time-efficiency of ML-based test prioritization in CI environments.
• To investigate how the length of test execution history used for training affects ML model performance.
• To analyze the impact of varying time budgets on the effectiveness of ML-based test prioritization approaches.
• To compare ML models (LambdaRank, SVM, GBDT, ANN) and heuristics (ROCKET, Random) in real-world CI settings.
• To provide empirical guidance on configuring ML models for optimal performance in industrial CI pipelines.

Proposed method

• The study uses real-world industrial datasets from Cisco, ABB, and Google for training and evaluating ML models.
• Four ML models—LambdaRank (learning-to-rank), SVM, GBDT, and ANN—are trained using test execution histories of varying lengths (from 10% to 100% of available data).
• Test case prioritization is evaluated using APFD (Average Precision at Fault Detection) to measure fault-detection effectiveness.
• Time-effectiveness is measured via TDFF (time to detect first fault), TDLF (time to detect last fault), TRAIN (model training time), and PART (prioritization algorithm runtime).
• The models are compared across different time budget levels (B1–B5) representing varying CI cycle constraints.
• Experiments are conducted on three industrial datasets to ensure real-world relevance and generalizability.

Figure 1: An iteration of CI testing consists of a code commit, build, and test phase, before code deployment. Change impact analysis and regression test prioritization are used to speed up testing. Regression test prioritization can use test history to learn to prioritize test cases and ML to scale

Experimental results

Research questions

• RQ1 How does the length of test execution history used for training affect the fault-prediction performance of ML-based test prioritization models?
• RQ2 How do different ML models perform under varying time budgets in CI cycles?
• RQ3 Which ML model achieves the best balance between fault-detection effectiveness and time-efficiency in CI testing?
• RQ4 How do ML-based approaches compare to heuristic baselines (ROCKET and Random) in terms of fault detection and runtime?

Key findings

• The optimal test history length for training ML models lies between 40% and 60% of available execution history, as performance peaks in this range.
• The ANN model is least sensitive to older test history, showing stable performance across different history lengths, but performs worst with short histories.
• For short time budgets, LambdaRank (LRN) achieves the highest fault-detection effectiveness, with the best APFD scores, outperforming all other ML models.
• The ANN model has the worst fault-detection performance under short time budgets, despite its fast ranking time.
• In terms of time-to-first-fault (TDFF), LambdaRank and SVM show the best performance, while Random performs worst.
• The ANN model has the fastest ranking time (PART), followed by GBDT and LRN, while Random is the fastest due to its simplicity.

Figure 2: Test history consisting of 15 CI cycles. Cycle 1 is the most recent and has the highest weight, while cycle 15 is the oldest and has the lowest wight. Test cases can change execution result between pass and fail in consecutive executions (CI cycles). Red: fail, Green: pass, Grey: inconclus