[Paper Review] Black-Box Testing of Deep Neural Networks Through Test Case Diversity

Zohreh Aghababaeyan, Manel Abdellatif+3 more|arXiv (Cornell University)|Dec 20, 2021

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes geometric diversity as a black-box, input-level metric to guide deep neural network (DNN) testing without requiring access to model internals or training data. It demonstrates that geometric diversity outperforms state-of-the-art white-box coverage criteria in predicting fault detection capability and reduces computational cost, making it a more effective and practical surrogate for test adequacy in image-based DNNs.

ABSTRACT

Deep Neural Networks (DNNs) have been extensively used in many areas including image processing, medical diagnostics, and autonomous driving. However, DNNs can exhibit erroneous behaviours that may lead to critical errors, especially when used in safety-critical systems. Inspired by testing techniques for traditional software systems, researchers have proposed neuron coverage criteria, as an analogy to source code coverage, to guide the testing of DNN models. Despite very active research on DNN coverage, several recent studies have questioned the usefulness of such criteria in guiding DNN testing. Further, from a practical standpoint, these criteria are white-box as they require access to the internals or training data of DNN models, which is in many contexts not feasible or convenient. In this paper, we investigate black-box input diversity metrics as an alternative to white-box coverage criteria. To this end, we first select and adapt three diversity metrics and study, in a controlled manner, their capacity to measure actual diversity in input sets. We then analyse their statistical association with fault detection using four datasets and five DNN models. We further compare diversity with state-of-the-art white-box coverage criteria. Our experiments show that relying on the diversity of image features embedded in test input sets is a more reliable indicator than coverage criteria to effectively guide the testing of DNNs. Indeed, we found that one of our selected black-box diversity metrics far outperforms existing coverage criteria in terms of fault-revealing capability and computational time. Results also confirm the suspicions that state-of-the-art coverage metrics are not adequate to guide the construction of test input sets to detect as many faults as possible with natural inputs.

Motivation & Objective

• To address the limitations of white-box coverage criteria in DNN testing, which require model internals and show weak correlation with fault detection using natural inputs.
• To develop a black-box testing approach that relies solely on input data diversity, avoiding execution or output analysis of the DNN under test.
• To evaluate whether input diversity metrics can serve as reliable surrogates for fault detection capability in DNNs.
• To compare the performance of diversity metrics against state-of-the-art coverage criteria in terms of fault detection and computational efficiency.
• To propose a clustering-based method for estimating fault detection that accounts for redundant mispredictions due to similar failure patterns.

Proposed method

• Adapted and evaluated three input diversity metrics—geometric diversity, feature-based diversity, and NCD-based diversity—on image-based DNN test sets.
• Used a controlled experimental setup with four datasets and five pre-trained DNN models to assess diversity and fault detection relationships.
• Proposed a clustering-based fault estimation technique to group mispredicted inputs by similarity, reducing overcounting of redundant faults.
• Measured the statistical association between diversity metrics and fault detection rates across multiple DNN architectures and datasets.
• Compared the performance of the best diversity metric against state-of-the-art coverage criteria (e.g., neuron coverage) in both fault detection correlation and execution time.
• Selected and reproduced results from prior work to ensure fair comparison with existing coverage-based approaches.

Experimental results

Research questions

• RQ1 Can black-box input diversity metrics serve as effective surrogates for fault detection in DNNs without requiring model internals?
• RQ2 How does geometric diversity compare to state-of-the-art white-box coverage criteria in predicting fault detection using natural test inputs?
• RQ3 To what extent does input diversity correlate with actual fault detection, and does it outperform coverage in computational efficiency?
• RQ4 Can a clustering-based approach improve fault estimation by identifying redundant mispredictions due to similar failure patterns?
• RQ5 Is there a significant statistical association between input diversity and the number of distinct faults detected in DNN test suites?

Key findings

• Geometric diversity emerged as the most effective black-box diversity metric, significantly outperforming existing white-box coverage criteria in predicting fault detection.
• The best-performing diversity metric showed a stronger statistical correlation with fault detection than any of the evaluated coverage criteria across all datasets and models.
• Geometric diversity achieved higher fault detection capability with substantially lower computational cost compared to coverage-based methods.
• The study confirmed that state-of-the-art coverage criteria have limited utility in guiding the selection of natural test inputs for effective fault detection.
• The clustering-based fault estimation method successfully reduced overcounting of faults by identifying groups of similar mispredictions, improving the accuracy of fault detection measurement.
• Results suggest that input diversity, particularly geometric diversity, is a more reliable and practical metric than coverage for black-box DNN testing in real-world scenarios.