[Paper Review] Adversarial Spheres

Justin Gilmer, Luke Metz+5 more|arXiv (Cornell University)|Jan 9, 2018

Adversarial Robustness in Machine Learning8 references47 citations

TL;DR

The paper studies adversarial examples in a high-dimensional concentric-spheres dataset, proving a bound between average distance to the nearest error and test error, and showing various models approach this bound, implying robustness hinges on reducing generalization error.

ABSTRACT

State of the art computer vision models have been shown to be vulnerable to small adversarial perturbations of the input. In other words, most images in the data distribution are both correctly classified by the model and are very close to a visually similar misclassified image. Despite substantial research interest, the cause of the phenomenon is still poorly understood and remains unsolved. We hypothesize that this counter intuitive behavior is a naturally occurring result of the high dimensional geometry of the data manifold. As a first step towards exploring this hypothesis, we study a simple synthetic dataset of classifying between two concentric high dimensional spheres. For this dataset we show a fundamental tradeoff between the amount of test error and the average distance to nearest error. In particular, we prove that any model which misclassifies a small constant fraction of a sphere will be vulnerable to adversarial perturbations of size $O(1/\sqrt{d})$. Surprisingly, when we train several different architectures on this dataset, all of their error sets naturally approach this theoretical bound. As a result of the theory, the vulnerability of neural networks to small adversarial perturbations is a logical consequence of the amount of test error observed. We hope that our theoretical analysis of this very simple case will point the way forward to explore how the geometry of complex real-world data sets leads to adversarial examples.

Motivation & Objective

• Motivate and understand adversarial examples through a simple, well-defined high-dimensional dataset.
• Define and relate two fundamental measures of the error set: the error rate mu(E) and the average distance to the nearest error d(E).
• Demonstrate that small classification error implies proximity of most data points to errors in high dimensions.
• Provide theoretical bounds on adversarial robustness independent of model architecture.
• Show that practical neural networks align with the theoretical bound on this synthetic task.

Proposed method

• Study a two-sphere data distribution in n dimensions (concentric spheres with radii 1 and R=1.3).
• Train various neural network architectures on this dataset and evaluate both test error and proximity to errors using a data-manifold adversarial attack (manifold PGD with constraint ||x||2 fixed).
• Formulate and analyze an analytically tractable quadratic network whose decision boundary is an ellipsoid to derive conditions for presence/absence of adversarial examples.
• Prove a bound: d(E) <= O(Phi^{-1}(p)/sqrt(n)) where p is inner-sphere accuracy and E is misclassified points on the inner sphere.
• Show that neural networks with different architectures approach the bound observed in practice across training set sizes N.
• Use Central Limit Theorem-based estimates to relate alpha_i parameters in the quadratic network to estimated error rates.

Experimental results

Research questions

• RQ1 What is the relationship between the error rate on the data manifold (mu(E)) and the average distance to the nearest error (d(E)) in high dimensions?
• RQ2 Can a simple high-dimensional synthetic task reveal fundamental bounds on adversarial robustness independent of model architecture?
• RQ3 Do different neural network architectures exhibit decision boundaries that conform to theoretical bounds on d(E) given mu(E)?
• RQ4 Is improving adversarial robustness possible without reducing test error on this dataset?

Key findings

• There exist models that correctly classify most randomly chosen points yet have nearby misclassifications (adversarial examples) on the data manifold.
• For this dataset, any model misclassifying a small constant fraction of the inner sphere will have adversarial perturbations of size O(1/√n).
• Neural networks of various architectures trained on the dataset approach the theoretical bound linking mu(E) and d(E).
• A quadratic network with a twinned analytic form demonstrates that imperfect parameter alignment (alpha_i outside [1/R^2,1]) leads to adversarial examples, despite tiny empirical test error.
• The bound on d(E) can be estimated and is tight relative to observed mu(E); improving robustness requires a substantial decrease in mu(E).
• The observed relationship between mu(E) and d(E) is similar across different architectures, suggesting a geometry-driven bound rather than architecture-specific phenomena.