[Paper Review] Testing Robustness Against Unforeseen Adversaries

Kaufmann, Max, Daniel Kang+10 more|arXiv (Cornell University)|Aug 21, 2019

Adversarial Robustness in Machine Learning56 references82 citations

TL;DR

The paper introduces ImageNet-UA, a benchmark and UA2 metric to evaluate unforeseen robustness against 19 non-Lp adversaries, and shows that unseen attacks expose weaknesses not captured by traditional Lp robustness.

ABSTRACT

Adversarial robustness research primarily focuses on L_p perturbations, and most defenses are developed with identical training-time and test-time adversaries. However, in real-world applications developers are unlikely to have access to the full range of attacks or corruptions their system will face. Furthermore, worst-case inputs are likely to be diverse and need not be constrained to the L_p ball. To narrow in on this discrepancy between research and reality we introduce ImageNet-UA, a framework for evaluating model robustness against a range of unforeseen adversaries, including eighteen new non-L_p attacks. To perform well on ImageNet-UA, defenses must overcome a generalization gap and be robust to a diverse attacks not encountered during training. In extensive experiments, we find that existing robustness measures do not capture unforeseen robustness, that standard robustness techniques are beat by alternative training strategies, and that novel methods can improve unforeseen robustness. We present ImageNet-UA as a useful tool for the community for improving the worst-case behavior of machine learning systems.

Motivation & Objective

• Motivate evaluation of model robustness against a diverse, unseen set of adversaries beyond Lp-bounded attacks.
• Propose ImageNet-UA as a standardized benchmark for unforeseen robustness on ImageNet (and CIFAR-10-UA as a smaller analogue).
• Define the Unforeseen Adversarial Accuracy (UA2) metric to quantify performance across diverse attack types.
• Provide a repository of non-Lp differentiable attacks to enable broad robustness testing.
• Empirically analyze how existing defenses perform under unforeseen attacks and identify promising training/defense strategies.

Proposed method

• Define a general adversarial framework where attacks A modify inputs via a differentiable function A(x,δ) with latent perturbations δ.
• Develop 18 novel non-Lp attacks (plus Elastic as non-novel), each differentiable and bounded by an Lp constraint ε, enabling dataset-agnostic plug-and-play use.
• Use Projected Gradient Descent (PGD) to solve the inner minimization for each attack within its perturbation set Sx^A,ε.
• Construct ImageNet-UA and CIFAR-10-UA benchmarks by applying the attacks to standard datasets and evaluating classifications.
• Introduce UA2 as the average over a finite adversary population A of accuracy under attack A at level ε_A.
• Provide analysis comparing UA2 to existing robustness metrics and exploring training strategies that improve unforeseen robustness.

Experimental results

Research questions

• RQ1 How does unseen, non-Lp adversarial stress affect model accuracy compared to traditional Lp attacks?
• RQ2 Is unforeseen robustness captured by existing metrics, or does UA2 reveal distinct weaknesses?
• RQ3 Which training or defense strategies improve robustness to unforeseen attacks beyond Lp-targeted defenses?
• RQ4 To what extent do advances in standard CV benchmarks correlate with improved unforeseen robustness?
• RQ5 Can combining augmentation and Lp training or multi-attack strategies enhance unforeseen robustness across diverse models?

Key findings

• Unforeseen robustness (UA2) is distinct from traditional Lp robustness and cannot be fully predicted by existing metrics.
• L2-based adversarial training provides stronger improvements for unforeseen robustness (UA2) than L∞ training across several models.
• Combining image augmentations (e.g., PixMix) with L∞ training significantly boosts UA2 beyond either technique alone.
• Multi-attack robust training and perceptual-based defenses (PAT) also improve UA2, sometimes outperforming pure Lp baselines.
• Progress on modern CV models and training (scale, pretraining, augmentation) correlates with improved UA2, suggesting CV advances partially track unforeseen robustness.
• Optimized non-Lp attacks show that models with high L∞ robustness may still falter under unforeseen adversaries, highlighting the need for diversified evaluation.