[Paper Review] Constrained Adversarial Learning for Automated Software Testing: a literature review

João Vitorino, Tiago Dias+3 more|arXiv (Cornell University)|Mar 14, 2023

Adversarial Robustness in Machine Learning4 citations

TL;DR

This systematic review proposes integrating constrained adversarial learning into automated software testing to improve test coverage and robustness by generating functionally valid, attack-like inputs that respect input constraints. By leveraging GANs—particularly WGAN-GP—trained on execution path data, the method enhances branch coverage more efficiently than random testing, reducing manual effort and computational cost in white-box testing scenarios.

ABSTRACT

It is imperative to safeguard computer applications and information systems against the growing number of cyber-attacks. Automated software testing tools can be developed to quickly analyze many lines of code and detect vulnerabilities by generating function-specific testing data. This process draws similarities to the constrained adversarial examples generated by adversarial machine learning methods, so there could be significant benefits to the integration of these methods in testing tools to identify possible attack vectors. Therefore, this literature review is focused on the current state-of-the-art of constrained data generation approaches applied for adversarial learning and software testing, aiming to guide researchers and developers to enhance their software testing tools with adversarial testing methods and improve the resilience and robustness of their information systems. The found approaches were systematized, and the advantages and limitations of those specific for white-box, grey-box, and black-box testing were analyzed, identifying research gaps and opportunities to automate the testing tools with data generated by adversarial attacks.

Motivation & Objective

• Address the growing need for automated, efficient, and comprehensive software testing in the face of increasing cyber threats and complex systems.
• Identify limitations in current automated testing tools, particularly their reliance on manual test design, random input generation, or exhaustive parameter mutation.
• Explore how constrained data generation in adversarial machine learning can be adapted to improve software testing by generating valid, targeted test inputs.
• Guide researchers and practitioners in enhancing software testing tools through adversarial learning techniques that respect functional and structural constraints of the system under test.
• Highlight research gaps and opportunities for future work in applying adversarial methods to software testing, especially in white-box and grey-box testing contexts.

Proposed method

• Conducted a systematic literature review using the PRISMA framework to identify relevant studies on constrained data generation in adversarial machine learning and software testing.
• Searched bibliographic databases using broad, flexible search terms including variations of 'constrained data generation', 'adversarial examples', 'conditional generators', and 'perturbation constraints' to ensure comprehensive coverage.
• Focused on adversarial learning methods that generate perturbations respecting functional or syntactic constraints of software inputs, such as data type, format, or logical bounds.
• Evaluated white-box approaches that use execution path information (e.g., from Gcov) to train GANs to generate inputs that maximize code coverage by targeting untested branches.
• Applied Generative Adversarial Networks (GANs), particularly WGAN-GP, to learn the behavior of the system under test and generate new test inputs that improve branch coverage iteratively.
• Integrated feedback loops where generated inputs are executed, their execution paths logged, and the results used to retrain the GAN, enabling adaptive and coverage-driven test data generation.

Figure 1: PRISMA search process for RQ1.

Experimental results

Research questions

• RQ1 How can adversarial learning methods be leveraged for constrained data generation in software testing?
• RQ2 How can automated software testing tools benefit from constrained data generation techniques derived from adversarial machine learning?
• RQ3 What are the current limitations of existing automated testing tools in achieving high test coverage efficiently?
• RQ4 In what ways can GAN-based approaches improve test coverage compared to random or heuristic-based testing strategies?
• RQ5 What are the transferable techniques from adversarial machine learning that can be adapted to enhance software testing robustness and efficiency?

Key findings

• The WGAN-GP-based approach improved branch coverage in 5 out of 7 unit test functions compared to random testing, demonstrating the effectiveness of constraint-aware adversarial generation.
• White-box testing using execution path feedback from tools like Gcov enables iterative refinement of test inputs, significantly increasing coverage without exhaustive search.
• Most current automated testing tools rely on heuristic or random input mutation, which is computationally expensive and often fails to explore complex or constrained input spaces efficiently.
• Constrained adversarial learning reduces the search space by generating only valid, functionally plausible inputs, making testing more efficient and scalable for complex systems.
• There is a clear research gap in adapting adversarial machine learning techniques—especially GANs with constraint enforcement—for direct integration into software testing pipelines.
• Future integration of NLP to extract constraints from code and fuzzy logic with reinforcement learning for black-box API testing could further enhance adaptability and robustness of automated testing tools.

Figure 2: PRISMA search process for RQ2.