[Paper Review] Exploring the Landscape of Spatial Robustness

Logan Engstrom, Brandon Tran+3 more|arXiv (Cornell University)|Dec 7, 2017

Land Use and Ecosystem ServicesEnvironmental Science253 citations

TL;DR

The paper studies neural classifier brittleness to small rotations and translations, shows first-order methods struggle to find worst-case spatial perturbations, and proposes robust optimization and input-aggregation defenses that significantly improve spatial robustness.

ABSTRACT

The study of adversarial robustness has so far largely focused on perturbations bound in p-norms. However, state-of-the-art models turn out to be also vulnerable to other, more natural classes of perturbations such as translations and rotations. In this work, we thoroughly investigate the vulnerability of neural network--based classifiers to rotations and translations. While data augmentation offers relatively small robustness, we use ideas from robust optimization and test-time input aggregation to significantly improve robustness. Finally we find that, in contrast to the p-norm case, first-order methods cannot reliably find worst-case perturbations. This highlights spatial robustness as a fundamentally different setting requiring additional study. Code available at https://github.com/MadryLab/adversarial_spatial and https://github.com/MadryLab/spatial-pytorch.

Motivation & Objective

• Assess how small rotations and translations degrade image classifier accuracy across MNIST, CIFAR-10, and ImageNet.
• Characterize the optimization landscape for spatial perturbations and compare attack methods (first-order, grid, worst-of-k).
• Develop and evaluate defenses to improve robustness to spatial transformations.
• Examine the interaction between spatial perturbations and traditional pixel-based perturbations.
• Propose practical strategies (robust training and input aggregation) to enhance spatial robustness.

Proposed method

• Parametrize spatial perturbations with rotation angle and pixel translations.
• Attack via: (i) first-order optimization in rotation-translation parameter space, (ii) grid search over the perturbation space, (iii) worst-of-k sampling over perturbations.
• Use differentiable spatial transformer networks to implement T(x; delta_u, delta_v, theta) with bilinear interpolation.
• Apply robust optimization-inspired training by adversarially selecting the worst of k perturbations per example during training.
• Propose inference-time aggregation by majority voting over multiple random spatialTransformations.
• Evaluate on MNIST, CIFAR-10, and ImageNet with constrained perturbation space (up to ~30 degrees and ~10% translation).
• Compare defenses including standard training, random rotations/translations, and worst-of-10 training to baseline and l_infinity adversarial training.

Experimental results

Research questions

• RQ1 How do small rotations and translations affect classifier accuracy across common vision datasets?
• RQ2 Are first-order optimization methods effective for finding worst-case spatial perturbations compared to grid search or random sampling?
• RQ3 Can robust training or input-aggregation strategies mitigate spatial brittleness?
• RQ4 Do spatial perturbations interact with pixel-based (l_infty) perturbations, and are robustness notions orthogonal?

Key findings

• Classifiers are brittle to spatial perturbations; small random rotations/translations can reduce accuracy by large margins (e.g., substantial drops on MNIST, CIFAR-10, and ImageNet).
• Exhaustive grid search-based adversaries are more powerful than first-order methods for spatial perturbations, due to a highly non-concave loss landscape.
• Worst-of-10 random spatial perturbations is a strong black-box-like attack, achieving large accuracy reductions with limited queries.
• Data augmentation with random spatial transforms improves robustness on MNIST and CIFAR-10 but less so on ImageNet; l_infinity adversarial training is not sufficient for spatial robustness.
• Robust optimization (worst-of-10 training) and majority-inference (aggregation of random transforms) significantly boost spatial robustness, yielding substantial gains over standard training and augmentation alone.
• Combined spatial and l_infinity perturbations have largely orthogonal effects, with robustness improving cumulatively when defending against both.