[Paper Review] Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models

Wieland Brendel, Jonas Rauber+1 more|arXiv (Cornell University)|Dec 12, 2017

Adversarial Robustness in Machine LearningComputer Science889 citations

TL;DR

The Boundary Attack is a simple, effective decision-based adversarial attack that works on black-box models by starting from a large adversarial perturbation and iteratively shrinking it along the decision boundary, performing comparably to gradient-based attacks on standard vision tasks.

ABSTRACT

Many machine learning algorithms are vulnerable to almost imperceptible perturbations of their inputs. So far it was unclear how much risk adversarial perturbations carry for the safety of real-world machine learning applications because most methods used to generate such perturbations rely either on detailed model information (gradient-based attacks) or on confidence scores such as class probabilities (score-based attacks), neither of which are available in most real-world scenarios. In many such cases one currently needs to retreat to transfer-based attacks which rely on cumbersome substitute models, need access to the training data and can be defended against. Here we emphasise the importance of attacks which solely rely on the final model decision. Such decision-based attacks are (1) applicable to real-world black-box models such as autonomous cars, (2) need less knowledge and are easier to apply than transfer-based attacks and (3) are more robust to simple defences than gradient- or score-based attacks. Previous attacks in this category were limited to simple models or simple datasets. Here we introduce the Boundary Attack, a decision-based attack that starts from a large adversarial perturbation and then seeks to reduce the perturbation while staying adversarial. The attack is conceptually simple, requires close to no hyperparameter tuning, does not rely on substitute models and is competitive with the best gradient-based attacks in standard computer vision tasks like ImageNet. We apply the attack on two black-box algorithms from Clarifai.com. The Boundary Attack in particular and the class of decision-based attacks in general open new avenues to study the robustness of machine learning models and raise new questions regarding the safety of deployed machine learning systems. An implementation of the attack is available as part of Foolbox at https://github.com/bethgelab/foolbox .

Motivation & Objective

• Highlight the relevance of decision-based attacks for real-world black-box models.
• Introduce the Boundary Attack as the first effective decision-based method for complex datasets.
• Show that decision-based attacks can break certain defensive strategies.
• Demonstrate applicability to real-world black-box APIs (Clarifai) and standard vision benchmarks.

Proposed method

• Propose a boundary-following attack that starts from an adversarial example and performs rejection sampling to move along the boundary toward minimal perturbations.
• Use a simple proposal distribution: sample Gaussian directions, project onto a sphere, and move toward the original input with two adjustable step sizes (orthogonal and toward-origin).
• Allow arbitrary adversarial criteria and only require final-model decision, not confidences or gradients.
• Dynamically adjust perturbation length and step size using a Trust-Region-inspired scheme based on the local boundary geometry.
• Evaluate in untargeted and targeted settings across MNIST, CIFAR-10, and ImageNet using standard architectures (VGG-19, ResNet-50, Inception-v3).
• Compare against gradient-based attacks (FGSM, DeepFool, Carlini & Wagner) in terms of perturbation size and robustness to defenses.

Experimental results

Research questions

• RQ1 Can a decision-based attack reliably generate adversarial examples for complex, real-world models without access to gradients or confidence scores?
• RQ2 How does the Boundary Attack perform in untargeted and targeted scenarios on MNIST, CIFAR-10, and ImageNet relative to gradient-based methods?
• RQ3 Is the Boundary Attack robust to defenses such as gradient masking and defensive distillation?
• RQ4 Can the Boundary Attack operate effectively in black-box, real-world APIs (e.g., Clarifai) where only final decisions are observable?],
• RQ5 key_findings':['The Boundary Attack achieves competitive minimal perturbations compared with gradient-based attacks across MNIST, CIFAR, and ImageNet in untargeted settings.','In untargeted ImageNet experiments, Boundary (ours) attains a median perturbation measure of 3.6e-3 on MNIST, 5.6e-6 on CIFAR, 2.9e-7 on VGG-19, 1.0e-7 on ResNet-50, and 6.5e-8 on Inception-v3.','In targeted settings, Boundary (ours) yields perturbations of 6.5e-3 (MNIST), 3.3e-5 (CIFAR), and 9.9e-06 (ImageNet with VGG-19).','The Boundary Attack remains effective when defenses such as defensive distillation are applied, where gradient-based attacks fail or degrade, demonstrating immunity to gradient masking.','On two Clarifai black-box models (brand and celebrity recognition), the Boundary Attack could produce adversarial examples with perturbations typically around 1e-2 to 1e-3, though some samples required larger perturbations for misclassification.','The attack uses zero backward passes and significantly more forward passes than gradient-based attacks, reflecting its reliance on model decisions rather than gradients.'],
• RQ6 table_headers':['Attack Type','MNIST','CIFAR','VGG-19','ResNet-50','Inception-v3'],
• RQ7 table_rows**: [ [