[Paper Review] Simple Black-Box Adversarial Perturbations for Deep Networks

Nina Narodytska, Shiva Prasad Kasiviswanathan|arXiv (Cornell University)|Dec 19, 2016

Adversarial Robustness in Machine Learning14 references165 citations

TL;DR

The paper shows that deep CNNs are vulnerable to black-box adversarial perturbations, constructing misclassified images by perturbing very few pixels without access to model parameters. It introduces random-pixel attacks and a greedy local-search approach to generate adversarial examples under a black-box threat model.

ABSTRACT

Deep neural networks are powerful and popular learning models that achieve state-of-the-art pattern recognition performance on many computer vision, speech, and language processing tasks. However, these networks have also been shown susceptible to carefully crafted adversarial perturbations which force misclassification of the inputs. Adversarial examples enable adversaries to subvert the expected system behavior leading to undesired consequences and could pose a security risk when these systems are deployed in the real world. In this work, we focus on deep convolutional neural networks and demonstrate that adversaries can easily craft adversarial examples even without any internal knowledge of the target network. Our attacks treat the network as an oracle (black-box) and only assume that the output of the network can be observed on the probed inputs. Our first attack is based on a simple idea of adding perturbation to a randomly selected single pixel or a small set of them. We then improve the effectiveness of this attack by carefully constructing a small set of pixels to perturb by using the idea of greedy local-search. Our proposed attacks also naturally extend to a stronger notion of misclassification. Our extensive experimental results illustrate that even these elementary attacks can reveal a deep neural network's vulnerabilities. The simplicity and effectiveness of our proposed schemes mean that they could serve as a litmus test for designing robust networks.

Motivation & Objective

• Assess vulnerability of state-of-the-art CNNs to black-box adversarial attacks with oracle-only access.
• Demonstrate that perturbing a single pixel or a small set of pixels can cause misclassification.
• Develop and evaluate greedy local-search based attacks to reduce required perturbation.
• Extend attacks to k-misclassification where the true label is outside the top-k predictions.

Proposed method

• Treat the network as an oracle and observe outputs on probed inputs.
• Investigate perturbing a single pixel (or small set) with sign-preserving perturbations.
• Define critical pixels and critical sets that can cause misclassification when perturbed.
• Propose RandAdv: a random-pixel perturbation method to estimate the fraction of critical pixels.
• Develop a greedy local-search attack that perturbs few pixels to minimize the probability that the true label remains in the top-k predictions.
• Extend the approach to high-resolution images via perturbing sets of pixels (e.g., 50 pixels) and larger perturbations.

Experimental results

Research questions

• RQ1 Can a black-box adversary cause misclassification by perturbing only a single pixel or a very small set of pixels?
• RQ2 How does perturbation magnitude influence the existence and detectability of critical pixels?
• RQ3 Can a greedy local-search strategy generate effective adversarial examples with limited perturbations under black-box access?
• RQ4 Is it possible to achieve k-misclassification under a black-box threat model?

Key findings

• A single randomly chosen pixel perturbation can frequently cause misclassification for many datasets.
• Increasing the perturbation magnitude increases the fraction of critical pixels and the success rate of RandAdv.
• For high-resolution images, perturbing around 50 pixels can effectively generate adversarial examples.
• The greedy local-search approach yields adversarial images with small perturbations and does not require access to network gradients.
• The attacks can achieve k-misclassification, ensuring the true label falls outside the top-k predictions.
• On ImageNet1000, the approach perturbs only about 0.5% of pixels on average.