[Paper Review] Adversarial Weight Perturbation Helps Robust Generalization

Dongxian Wu, Shu‐Tao Xia+1 more|arXiv (Cornell University)|Apr 13, 2020

Adversarial Robustness in Machine Learning74 references206 citations

TL;DR

The paper introduces Adversarial Weight Perturbation (AWP), a regularizer that perturbs model weights adversarially to flatten the weight loss landscape and improve robustness in adversarial training.

ABSTRACT

The study on improving the robustness of deep neural networks against adversarial examples grows rapidly in recent years. Among them, adversarial training is the most promising one, which flattens the input loss landscape (loss change with respect to input) via training on adversarially perturbed examples. However, how the widely used weight loss landscape (loss change with respect to weight) performs in adversarial training is rarely explored. In this paper, we investigate the weight loss landscape from a new perspective, and identify a clear correlation between the flatness of weight loss landscape and robust generalization gap. Several well-recognized adversarial training improvements, such as early stopping, designing new objective functions, or leveraging unlabeled data, all implicitly flatten the weight loss landscape. Based on these observations, we propose a simple yet effective Adversarial Weight Perturbation (AWP) to explicitly regularize the flatness of weight loss landscape, forming a double-perturbation mechanism in the adversarial training framework that adversarially perturbs both inputs and weights. Extensive experiments demonstrate that AWP indeed brings flatter weight loss landscape and can be easily incorporated into various existing adversarial training methods to further boost their adversarial robustness.

Motivation & Objective

• Motivate robust generalization in adversarial training beyond input loss flattening.
• Characterize the relationship between weight loss landscape flatness and robust generalization gap.
• Propose and validate AWP to explicitly regularize weight loss landscape via double perturbations (inputs and weights).
• Demonstrate AWP's compatibility and robustness gains when integrated with existing adversarial training methods.

Proposed method

• Characterize weight loss landscape using on-the-fly adversarial examples generated by PGD.
• Show correlation between flat weight loss landscape and smaller robust generalization gap.
• Propose Adversarial Weight Perturbation (AWP) to maximize weight perturbation effects within a controlled region (γ||w_l||).
• Optimize a double-perturbation objective that alternates between adversarial input perturbations and adversarial weight perturbations.
• Provide an algorithm (AT-AWP) that updates a perturbed model f_{w+v} and then updates the center weights w accordingly.
• Extend AWP to other adversarial training frameworks (TRADES, MART, RST) with minimal overhead.

Experimental results

Research questions

• RQ1 Does the flatness of the weight loss landscape correlate with robust generalization gap under adversarial training?
• RQ2 Can explicit regularization of the weight loss landscape via adversarial weight perturbations improve robustness when combined with existing AT methods?
• RQ3 Is AWP compatible and beneficial across multiple datasets, architectures, and threat models?
• RQ4 How does AWP compare to random weight perturbations and other regularizations in improving adversarial robustness?

Key findings

• Flatter weight loss landscapes correlate with smaller robust generalization gaps during and across adversarial training methods.
• Adversarial Weight Perturbation (AWP) explicitly flattens the weight loss landscape and yields robustness gains when integrated with AT, TRADES, MART, and RST.
• AWP consistently improves test robustness across CIFAR-10/ SVHN/CIFAR-100 and L_infty and L2 threat models.
• AWP achieves improvements over baseline AT and comparable methods, with gains shown across white-box and black-box attacks, including AutoAttack.
• Ablation studies show that small, relative weight perturbations (gamma around 1e-3 to 5e-3) effectively flatten the landscape and reduce robust generalization gap.
• Compared to random weight perturbations, AWP provides stronger adversarial loss increases and better robustness with smaller perturbation magnitudes.