[Paper Review] Predicting Metamorphic Relation for Matrix Calculation Programs

Karishma Rahman, Upulee Kanewala|arXiv (Cornell University)|Feb 19, 2018

Software Testing and Debugging Techniques10 references3 citations

TL;DR

This paper proposes a graph kernel-based machine learning approach using random walk kernels on control flow graphs (CFGs) to automatically predict metamorphic relations (MRs) for matrix calculation programs. The method achieves high prediction accuracy, with an AUC of 0.81 for permutative MRs and above 0.7 for additive and multiplicative MRs, demonstrating that MRs in matrix programs can be effectively predicted without manual expert input.

ABSTRACT

Matrices often represent important information in scientific applications and are involved in performing complex calculations. But systematically testing these applications is hard due to the oracle problem. Metamorphic testing is an effective approach to test such applications because it uses metamorphic relations to determine whether test cases have passed or failed. Metamorphic relations are typically identified with the help of a domain expert and is a labor intensive task. In this work we use a graph kernel based machine learning approach to predict metamorphic relations for matrix calculation programs. Previously, this graph kernel based machine learning approach was used to successfully predict metamorphic relations for programs that perform numerical calculations. Results of this study show that this approach can be used to predict metamorphic relations for matrix calculation programs as well.

Motivation & Objective

• To address the oracle problem in testing scientific matrix calculation programs, which lack reliable test oracles.
• To automate the identification of metamorphic relations (MRs), a critical but labor-intensive step in metamorphic testing.
• To extend prior work on numerical programs to matrix computation functions, which involve complex data structures and operations.
• To evaluate whether graph kernel-based machine learning can effectively predict MRs for matrix-based functions.
• To assess the generalizability of random walk kernels for capturing MR-relevant patterns in matrix computation CFGs.

Proposed method

• Construct control flow graphs (CFGs) from matrix calculation functions, with nodes representing atomic operations and edges representing control flow.
• Apply post-processing to annotate method call nodes with return types and ensure atomic operation granularity.
• Compute similarity between CFGs using the random walk kernel, which measures similarity by summing over all pairs of walks in the graphs.
• Train a support vector machine (SVM) classifier on the kernel matrix to predict MRs, using 55 open-source matrix functions for training and testing.
• Optimize hyperparameters (C and λ) via 10-fold cross-validation using the validation set to prevent overfitting.
• Evaluate model performance using the Area Under the ROC Curve (AUC), which measures the probability of correct ranking between positive and negative MR examples.

Experimental results

Research questions

• RQ1 Can a graph kernel-based machine learning approach effectively predict metamorphic relations for matrix calculation programs?
• RQ2 How does the performance of the random walk kernel compare to other graph kernels in predicting MRs for matrix functions?
• RQ3 What hyperparameter settings (C and λ) yield the best predictive performance for different types of matrix-based MRs?
• RQ4 To what extent does the model generalize to unseen matrix functions, as indicated by validation and test set performance?
• RQ5 Are longer control flow paths in CFGs more informative for predicting MRs, as suggested by the optimal λ value?

Key findings

• The proposed method achieved an AUC of 0.81 for predicting permutative metamorphic relations, indicating strong predictive performance.
• AUC scores for additive and multiplicative MRs were both above 0.7, demonstrating consistent effectiveness across all three MR types.
• The optimal λ value of 0.9 for all MR types suggests that longer paths in the CFGs are more informative for MR prediction than shorter ones.
• The close alignment between validation and test AUC scores indicates low overfitting and good generalization of the predictive model.
• The choice of C value had minimal impact on performance, suggesting robustness across different regularization strengths.
• The results confirm that graph kernel-based learning can be successfully extended from numerical to matrix computation programs, enabling automated MR prediction.