[Paper Review] Adversarial Robustness through Local Linearization

Chongli Qin, James Martens+7 more|arXiv (Cornell University)|Jul 4, 2019

Adversarial Robustness in Machine LearningComputer Science26 references91 citations

TL;DR

Introduces a local linearity regularizer (LLR) to promote linear loss behavior near training data, enabling faster robust training and improved adversarial accuracy compared to standard adversarial training on CIFAR-10 and ImageNet.

ABSTRACT

Adversarial training is an effective methodology for training deep neural networks that are robust against adversarial, norm-bounded perturbations. However, the computational cost of adversarial training grows prohibitively as the size of the model and number of input dimensions increase. Further, training against less expensive and therefore weaker adversaries produces models that are robust against weak attacks but break down under attacks that are stronger. This is often attributed to the phenomenon of gradient obfuscation; such models have a highly non-linear loss surface in the vicinity of training examples, making it hard for gradient-based attacks to succeed even though adversarial examples still exist. In this work, we introduce a novel regularizer that encourages the loss to behave linearly in the vicinity of the training data, thereby penalizing gradient obfuscation while encouraging robustness. We show via extensive experiments on CIFAR-10 and ImageNet, that models trained with our regularizer avoid gradient obfuscation and can be trained significantly faster than adversarial training. Using this regularizer, we exceed current state of the art and achieve 47% adversarial accuracy for ImageNet with l-infinity adversarial perturbations of radius 4/255 under an untargeted, strong, white-box attack. Additionally, we match state of the art results for CIFAR-10 at 8/255.

Motivation & Objective

• Motivate and address the high computational cost of adversarial training for robust models.
• Propose a regularizer that enforces local linearity of the loss around training data to prevent gradient obfuscation.
• Demonstrate that Local Linearity Regularization (LLR) yields faster training and better or comparable robustness to strong attacks.
• Empirically evaluate LLR on CIFAR-10 and ImageNet against strong white-box adversaries and contrast with baselines like ADV, TRADES, and CURE.

Proposed method

• Define a local linearity measure gamma(epsilon, x) capturing the deviation from a first-order Taylor expansion within an epsilon-ball.
• Derive the Local Linearity Regularizer (LLR) that penalizes gamma(epsilon, x) and the inner perturbation term |delta_LL R^T grad_x ell(x), constrained to the epsilon-ball.
• Use an inner optimization to find delta_LL R via gradient descent, similar in spirit to adversarial training but typically with far fewer steps.
• Provide a combined objective L(D) = E[ ell(x) + lambda*gamma(epsilon, x) + mu*|delta_LL R^T grad ell(x)| ] to train robust models.
• Argue and empirically show that minimizing gamma(epsilon, x) suffices to bound adversarial loss and reduces gradient obfuscation.

Experimental results

Research questions

• RQ1 Can enforcing local linearity of the loss around training examples reduce gradient obfuscation and improve robustness to strong adversaries?
• RQ2 Is Local Linearity Regularization (LLR) faster to train than standard adversarial training while achieving or surpassing its robustness?
• RQ3 How does LLR perform on CIFAR-10 and ImageNet under strong untargeted and targeted white-box attacks compared to ADV, TRADES, and DENOISE?
• RQ4 What is the impact of LLR on robustness degradation when attackers increase strength of perturbations?

Key findings

• LLR yields state-of-the-art adversarial accuracy on CIFAR-10 at epsilon=8/255 and ImageNet at epsilon=4/255 under strong white-box attacks.
• Training with LLR is reported to be up to 5x faster on ImageNet compared to standard adversarial training.
• Models trained with LLR exhibit more graceful degradation in adversarial accuracy as attack strength increases than those trained with adversarial training.
• On ImageNet, LLR achieves 47% adversarial accuracy for epsilon=4/255 under an untargeted attack, outperforming several baselines.
• For CIFAR-10, LLR achieves 52.81% adversarial accuracy at epsilon=8/255, matching or exceeding the reported baselines under similar evaluation.