[Paper Review] Instance adaptive adversarial training: Improved accuracy tradeoffs in neural nets

Yogesh Balaji, Tom Goldstein+1 more|arXiv (Cornell University)|Oct 17, 2019

Adversarial Robustness in Machine LearningComputer Science21 references66 citations

TL;DR

The paper introduces instance adaptive adversarial training (IAAT), which assigns per-sample perturbation radii during training to improve clean accuracy with comparable robustness, demonstrated across CIFAR-10/100 and ImageNet.

ABSTRACT

Adversarial training is by far the most successful strategy for improving robustness of neural networks to adversarial attacks. Despite its success as a defense mechanism, adversarial training fails to generalize well to unperturbed test set. We hypothesize that this poor generalization is a consequence of adversarial training with uniform perturbation radius around every training sample. Samples close to decision boundary can be morphed into a different class under a small perturbation budget, and enforcing large margins around these samples produce poor decision boundaries that generalize poorly. Motivated by this hypothesis, we propose instance adaptive adversarial training -- a technique that enforces sample-specific perturbation margins around every training sample. We show that using our approach, test accuracy on unperturbed samples improve with a marginal drop in robustness. Extensive experiments on CIFAR-10, CIFAR-100 and Imagenet datasets demonstrate the effectiveness of our proposed approach.

Motivation & Objective

• Motivate the robustness-accuracy tradeoff in standard adversarial training and its limitations due to uniform perturbation radii.
• Propose IAAT to assign per-sample perturbation radii to better preserve clean accuracy while maintaining robustness.
• Show empirically that IAAT improves natural test accuracy with marginal robustness loss across datasets (CIFAR-10/100, ImageNet).
• Provide interpretability insights: small radii correlate with boundary ambiguity and nearby conflicting samples; large radii correlate with clear class.]
• compare IAAT to prior adaptive-margin work and analyze robustness across a range of test perturbations.

Proposed method

• Formulate IAAT as a min-max objective with per-sample perturbation budgets: min_theta max_{||delta_i||_infty <= epsilon_i} Loss( x_i + delta_i, y_i ).
• Initialize with a warmup period using a uniform epsilon before switching to per-sample epsilons.
• After crafting adversaries for each sample, adjust epsilon_i based on whether PGD succeeds (epsilon_i--) or fails (epsilon_i++), smoothed over epochs via a memory mechanism.
• Algorithm 1 provides the training loop that alternates between adversarial example generation and parameter updates.
• Algorithm 2 updates and stabilizes per-sample epsilon via a discretized search around prior values.
• The approach emphasizes selecting larger radii where class manifolds are well separated and smaller radii near decision boundaries, yielding better tradeoffs.
• Experiments include CIFAR-10/100 with ResNet and WideResNet architectures and ImageNet with ResNet variants, evaluating natural accuracy, white-box and transfer robustness, and corruption robustness.

Experimental results

Research questions

• RQ1 Does enforcing per-sample adversarial radii improve clean accuracy at fixed robustness levels compared to standard uniform-radius adversarial training?
• RQ2 Can IAAT break the traditional robustness-accuracy Pareto frontier observed in standard adversarial training?
• RQ3 How interpretable are the learned per-sample radii with respect to data ambiguity and class boundaries?
• RQ4 Do gains from IAAT generalize across datasets (CIFAR-10/100, ImageNet) and architectures?
• RQ5 What is the effect of warmup and per-sample radii dynamics on robustness to unseen corruptions?

Key findings

• IAAT improves natural test accuracy at comparable robustness to standard adversarial training on CIFAR-10/100 (examples: notable clean-accuracy gains with similar or marginally reduced adversarial robustness).
• IAAT yields robustness to unseen image corruptions, indicating improved generalization.
• On ImageNet, IAAT substantially raises natural accuracy (+10%+ over adversarial training) across ResNet models and maintains competitive adversarial robustness at lower perturbation strengths, with some tradeoffs at higher epsilons.
• Per-sample epsilons correlate with human notions of ambiguity: near-boundary samples receive smaller epsilons, while unambiguous samples receive larger ones.
• Warmup improves robustness with small drops in natural accuracy, and IAAT without warmup also shows improved tradeoffs, with warmup generally helping robustness more on CIFAR-100.
• IAAT can outperform Mixup-adversarial training on stronger attacks, maintaining robustness while preserving natural accuracy.