[Paper Review] Do Wider Neural Networks Really Help Adversarial Robustness?

Boxi Wu, Jinghui Chen+3 more|arXiv (Cornell University)|Oct 3, 2020

Adversarial Robustness in Machine Learning75 references35 citations

TL;DR

The paper analyzes how network width affects adversarial robustness under adversarial training, showing wider nets can hurt perturbation stability and proposing Width Adjusted Regularization (WAR) to mitigate this.

ABSTRACT

Adversarial training is a powerful type of defense against adversarial examples. Previous empirical results suggest that adversarial training requires wider networks for better performances. However, it remains elusive how neural network width affects model robustness. In this paper, we carefully examine the relationship between network width and model robustness. Specifically, we show that the model robustness is closely related to the tradeoff between natural accuracy and perturbation stability, which is controlled by the robust regularization parameter $λ$. With the same $λ$, wider networks can achieve better natural accuracy but worse perturbation stability, leading to a potentially worse overall model robustness. To understand the origin of this phenomenon, we further relate the perturbation stability with the network's local Lipschitzness. By leveraging recent results on neural tangent kernels, we theoretically show that wider networks tend to have worse perturbation stability. Our analyses suggest that: 1) the common strategy of first fine-tuning $λ$ on small networks and then directly use it for wide model training could lead to deteriorated model robustness; 2) one needs to properly enlarge $λ$ to unleash the robustness potential of wider models fully. Finally, we propose a new Width Adjusted Regularization (WAR) method that adaptively enlarges $λ$ on wide models and significantly saves the tuning time.

Motivation & Objective

• Investigate how network width influences model robustness in adversarial training.
• Characterize the trade-off between natural accuracy and perturbation stability.
• Theoretically connect perturbation stability to local Lipschitzness and neural tangent kernel (NTK) results.
• Propose methods to adapt regularization for wide models to improve robustness.
• Provide practical guidance on tuning regularization to unlock wider architectures' robustness potential.

Proposed method

• Formulate adversarial training with a robust regularization term controlled by lambda in the TRADES framework.
• Empirically measure natural accuracy, robust accuracy, and perturbation stability across widths using WideResNet on CIFAR-10.
• Analyze perturbation stability as a function of local Lipschitzness and network width.
• Use NTK-based arguments to show wider networks have larger local Lipschitz constants, degrading stability.
• Propose Width Adjusted Regularization (WAR) to adapt lambda for wider models and reduce tuning effort.

Experimental results

Research questions

• RQ1 How does network width affect perturbation stability and overall robustness under adversarial training?
• RQ2 What is the relationship between natural accuracy, perturbation stability, and robust accuracy as width increases?
• RQ3 Can larger robust regularization (lambda) mitigate width-induced degradation in stability and robustness?
• RQ4 Does a width-aware regularization strategy improve robustness without extensive hyperparameter tuning?

Key findings

• Wider networks often have better natural accuracy but worse perturbation stability, leading to no guaranteed robustness gain.
• Perturbation stability degrades with width due to increased local Lipschitz constants, and wider networks exhibit larger input-gradient norms.
• The trade-off is better described between natural accuracy and perturbation stability rather than natural vs robust accuracy alone.
• Increasing the robust regularization parameter lambda for wider models improves perturbation stability and robustness.
• A Width Adjusted Regularization (WAR) approach can adapt lambda with width, saving tuning time and boosting robustness across architectures and datasets.