[Paper Review] Detecting Adversarial Samples for Deep Neural Networks through Mutation Testing

Jingyi Wang, Jun Sun+2 more|arXiv (Cornell University)|May 14, 2018

Adversarial Robustness in Machine LearningComputer Science31 references43 citations

TL;DR

The paper proposes nMutant, a black-box adversarial sample detector for DNNs that uses mutation testing to measure sample sensitivity to random perturbations and detects adversarial inputs with statistical SPRT-based decisions. It shows higher sensitivity for adversarial samples compared to normal ones across MNIST and CIFAR-10, achieving notable detection accuracy with modest mutation counts.

ABSTRACT

Recently, it has been shown that deep neural networks (DNN) are subject to attacks through adversarial samples. Adversarial samples are often crafted through adversarial perturbation, i.e., manipulating the original sample with minor modifications so that the DNN model labels the sample incorrectly. Given that it is almost impossible to train perfect DNN, adversarial samples are shown to be easy to generate. As DNN are increasingly used in safety-critical systems like autonomous cars, it is crucial to develop techniques for defending such attacks. Existing defense mechanisms which aim to make adversarial perturbation challenging have been shown to be ineffective. In this work, we propose an alternative approach. We first observe that adversarial samples are much more sensitive to perturbations than normal samples. That is, if we impose random perturbations on a normal and an adversarial sample respectively, there is a significant difference between the ratio of label change due to the perturbations. Observing this, we design a statistical adversary detection algorithm called nMutant (inspired by mutation testing from software engineering community). Our experiments show that nMutant effectively detects most of the adversarial samples generated by recently proposed attacking methods. Furthermore, we provide an error bound with certain statistical significance along with the detection.

Motivation & Objective

• Motivate robust detection of adversarial samples without relying on known attack data.
• Show that adversarial inputs are more sensitive to random perturbations than normal inputs.
• Develop a practical, scalable detector using mutation testing with statistical guarantees.
• Provide guidance on parameter settings and demonstrate effectiveness on MNIST and CIFAR-10.

Proposed method

• Define mutation testing framework by perturbing input x to create labeled mutations x_m within realistic perturbation bounds.
• Measure sensitivity κ(x) as the fraction of mutations that change the model's output: κ(x) = |{x_m : f(x_m) ≠ f(x)}| / |X_m(x)|.
• Apply Sequential Probability Ratio Test (SPRT) to decide if κ(x) > μ·κ1 or κ(x) ≤ μ·κ1 with controlled error α, β.
• Use κ1 as an upper bound on normal-sample sensitivity, and μ, α, β, σ to control decision regions and confidence.
• Generate mutations with a StepSize that captures realistic perturbations and maximize separation between κnor and κadv.
• Provide a practical algorithm (Algorithm 1) that stops when a decision is reached with bounded error.

Experimental results

Research questions

• RQ1 Can mutation testing create a reliable signal to distinguish normal from adversarial samples without knowledge of the model internals?
• RQ2 What is the empirical difference between the sensitivity of normal and adversarial samples to random perturbations across common attacks?
• RQ3 How many mutations are needed to detect adversarial inputs with specified confidence in MNIST and CIFAR-10?
• RQ4 How do parameter choices (StepSize, κ1, μ, α, β) affect detection accuracy and efficiency?

Key findings

• Adversarial samples exhibit significantly higher mutation sensitivity κadv than normal samples κnor across FGSM, C&W, JSMA, and Black-box attacks.
• Smaller StepSize increases the distance between κnor and κadv, aiding earlier detection; larger StepSize reduces this distance.
• Detection accuracy averages 77.4% on MNIST and 76.6% on CIFAR-10, with about 33 mutations (MNIST) and 46.5 mutations (CIFAR-10) per sample.
• Detection also identifies wrongly-labeled samples effectively, suggesting potential for relabeling such data.
• Normal samples are identified correctly with high precision, averaging about 97% accuracy.
• The detector operates in a black-box fashion and provides a confidence-bound decision for each input.