[Paper Review] The Space of Transferable Adversarial Examples

Florian Tramèr, Nicolas Papernot+3 more|arXiv (Cornell University)|Apr 11, 2017

Adversarial Robustness in Machine LearningComputer Science20 references435 citations

TL;DR

The paper estimates the dimensionality of adversarial subspaces and shows that transferable adversarial examples occupy a high-dimensional, overlapping space across models, with boundaries that are close across diverse architectures.

ABSTRACT

Adversarial examples are maliciously perturbed inputs designed to mislead machine learning (ML) models at test-time. They often transfer: the same adversarial example fools more than one model. In this work, we propose novel methods for estimating the previously unknown dimensionality of the space of adversarial inputs. We find that adversarial examples span a contiguous subspace of large (~25) dimensionality. Adversarial subspaces with higher dimensionality are more likely to intersect. We find that for two different models, a significant fraction of their subspaces is shared, thus enabling transferability. In the first quantitative analysis of the similarity of different models' decision boundaries, we show that these boundaries are actually close in arbitrary directions, whether adversarial or benign. We conclude by formally studying the limits of transferability. We derive (1) sufficient conditions on the data distribution that imply transferability for simple model classes and (2) examples of scenarios in which transfer does not occur. These findings indicate that it may be possible to design defenses against transfer-based attacks, even for models that are vulnerable to direct attacks.

Motivation & Objective

• Quantify the dimensionality of adversarial subspaces and their transferability across models.
• Assess how close different models' decision boundaries are in adversarial and benign directions.
• Investigate conditions under which transferability occurs or fails and analyze the impact of defenses like adversarial training.

Proposed method

• Introduce Gradient Aligned Adversarial Subspace (GAAS) to find multiple orthogonal adversarial directions.
• Use first-order loss approximations to generate and count orthogonal perturbations within a given norm bound.
• Measure transferability by testing perturbations on source and target models across MNIST and DREBIN datasets.
• Analyze inter-boundary and minimum distances in legitimate, adversarial, and random directions to compare model boundaries.
• Examine model-agnostic perturbations based on class-mean differences and provide theoretical conditions for transferability.

Experimental results

Research questions

• RQ1 What is the effective dimensionality of the adversarial subspace that fools neural networks and other models?
• RQ2 How similar are the decision boundaries of different models, particularly in adversarial directions, and how does this relate to transferability?
• RQ3 Under what data distributions and model classes can transferability be guaranteed or fail, and what are the sufficient conditions for transferability?
• RQ4 How do defenses like adversarial training affect the proximity of decision boundaries and the practicality of black-box attacks?

Key findings

• Adversarial examples span a contiguous, multi-dimensional subspace; e.g., two fully-connected networks on MNIST yield a 25-dimensional transferable subspace with about 24.87 directions transferring to the target model.
• Randomly sampling within the spanned subspace misclassifies the source model in 99% of cases and the target model in 89% of cases for MNIST CNNs/FCs; transfer rates vary by model pair (e.g., 68% for CNNs on MNIST).
• Decision boundaries of different models lie very close in both adversarial and benign directions, indicating high boundary similarity across model classes.
• Adversarial training increases inter-boundary distances but does not fully prevent transferability; transferred perturbations can still overshoot the source boundary and fool defended models.
• Model-agnostic perturbations based on class-mean differences can transfer to linear and quadratic models under certain alignment conditions; transfer can fail when such alignment and feature mappings are not preserved (XOR artifact example).
• The paper provides sufficient conditions for transferability in simple model classes and also presents a counterexample where transferability does not hold, showing it is not universal across all setups.