[Paper Review] Exploring the Space of Black-box Attacks on Deep Neural Networks

Arjun Nitin Bhagoji, Warren He+2 more|arXiv (Cornell University)|Dec 27, 2017

Adversarial Robustness in Machine LearningComputer Science24 references70 citations

TL;DR

The paper introduces Gradient Estimation black-box attacks that use model query outputs to craft adversarial examples, achieving near white-box performance on MNIST and CIFAR-10 and beating transfer-based black-box methods, with effective query reduction strategies.

ABSTRACT

Existing black-box attacks on deep neural networks (DNNs) so far have largely focused on transferability, where an adversarial instance generated for a locally trained model can "transfer" to attack other learning models. In this paper, we propose novel Gradient Estimation black-box attacks for adversaries with query access to the target model's class probabilities, which do not rely on transferability. We also propose strategies to decouple the number of queries required to generate each adversarial sample from the dimensionality of the input. An iterative variant of our attack achieves close to 100% adversarial success rates for both targeted and untargeted attacks on DNNs. We carry out extensive experiments for a thorough comparative evaluation of black-box attacks and show that the proposed Gradient Estimation attacks outperform all transferability based black-box attacks we tested on both MNIST and CIFAR-10 datasets, achieving adversarial success rates similar to well known, state-of-the-art white-box attacks. We also apply the Gradient Estimation attacks successfully against a real-world Content Moderation classifier hosted by Clarifai. Furthermore, we evaluate black-box attacks against state-of-the-art defenses. We show that the Gradient Estimation attacks are very effective even against these defenses.

Motivation & Objective

• Motivate and quantify the threat of black-box adversaries with query access to model outputs without relying on transferability.
• Develop Gradient Estimation-based attacks that approximate gradients via finite differences.
• Reduce the number of queries needed to craft adversarial samples while maintaining high success rates.
• Evaluate attacks against state-of-the-art models, defenses, and real-world APIs (Clarifai).
• Provide a comprehensive comparison against zero-query baselines and transfer-based black-box attacks.

Proposed method

• Propose Gradient Estimation attacks that approximate gradients using finite differences with access to the target model’s output probabilities.
• Use two query-reduction techniques: random feature grouping and PCA-based query reduction to lower dimensionality.
• Experiment with both cross-entropy loss and a logit-based loss to guide perturbations.
• Develop single-step and iterative variants (IFD-xent, IFD-logit, IFD-xent-T, IFD-logit-T) to achieve high success rates.
• Compare against zero-query baselines, transfer-based attacks, and white-box baselines on MNIST and CIFAR-10.
• Demonstrate applicability to a real-world API (Clarifai NSFW/Content Moderation) and defenses.

Experimental results

Research questions

• RQ1 Can black-box adversaries achieve high attack success rates without training local surrogate models?
• RQ2 How effectively can gradient information be estimated from query outputs to generate adversarial examples?
• RQ3 Do query-reduction strategies make high-dimensional gradient estimation practical for images?
• RQ4 Are Gradient Estimation attacks robust against modern defenses and real-world MLaaS services?

Key findings

• Gradient Estimation attacks match white-box attack performance on MNIST and CIFAR-10 for single-step and iterative settings.
• Iterative Gradient Estimation attacks (IFD-*) achieve 100% adversarial success across models and datasets.
• Single-step Gradient Estimation with logit loss (FD-logit) often matches white-box FGSM with logit loss, requiring far fewer queries (roughly 200–800 per image for single-step; ~8,000 for iterative).
• Query-reduction methods (random grouping, PCA-based) substantially reduce the number of queries with little loss in success rate.
• Attacks remain highly effective against standard, ensemble, and iterative adversarial training defenses, and against real-world Clarifai NSFW/Content Moderation models (demonstrated with ~200 queries per image).
• Compared to transfer-based black-box attacks, Gradient Estimation approaches significantly outperform in attack success with comparable distortions.