[论文解读] Do Wider Neural Networks Really Help Adversarial Robustness?

Boxi Wu, Jinghui Chen+3 more|arXiv (Cornell University)|Oct 3, 2020

Adversarial Robustness in Machine Learning参考文献 75被引用 35

一句话总结

该论文分析网络宽度在对抗训练下对对抗鲁棒性的影响，指出更宽的网络可能削弱扰动稳定性，并提出宽度自适应正则化（Width Adjusted Regularization，WAR）以缓解这一问题。

ABSTRACT

Adversarial training is a powerful type of defense against adversarial examples. Previous empirical results suggest that adversarial training requires wider networks for better performances. However, it remains elusive how neural network width affects model robustness. In this paper, we carefully examine the relationship between network width and model robustness. Specifically, we show that the model robustness is closely related to the tradeoff between natural accuracy and perturbation stability, which is controlled by the robust regularization parameter $λ$. With the same $λ$, wider networks can achieve better natural accuracy but worse perturbation stability, leading to a potentially worse overall model robustness. To understand the origin of this phenomenon, we further relate the perturbation stability with the network's local Lipschitzness. By leveraging recent results on neural tangent kernels, we theoretically show that wider networks tend to have worse perturbation stability. Our analyses suggest that: 1) the common strategy of first fine-tuning $λ$ on small networks and then directly use it for wide model training could lead to deteriorated model robustness; 2) one needs to properly enlarge $λ$ to unleash the robustness potential of wider models fully. Finally, we propose a new Width Adjusted Regularization (WAR) method that adaptively enlarges $λ$ on wide models and significantly saves the tuning time.

研究动机与目标

• 调查网络宽度如何影响对抗训练中的模型鲁棒性。
• 表征自然准确度与扰动稳定性之间的权衡。
• 在理论上将扰动稳定性与局部Lipschitz性及神经切线核（NTK）结果联系起来。
• 提出适应宽模型的正则化以提升鲁棒性的方法。
• 提供关于调节正则化以释放更宽架构鲁棒潜力的实用指南。

提出的方法

• 在 TRADES 框架中，使用受 lambda 控制的鲁棒正则化项来表述对抗训练。
• 使用 CIFAR-10 的 WideResNet，在不同宽度下实证测量自然准确度、鲁棒准确度和扰动稳定性。
• 将扰动稳定性作为局部Lipschitzness和网络宽度的函数进行分析。
• 基于 NTK 的论点显示更宽的网络具有更大的局部 Lipschitz 常数，从而降低稳定性。
• 提出 Width Adjusted Regularization (WAR) 以使 lambda 适应更宽的模型并减少调参工作。

实验结果

研究问题

• RQ1 在对抗训练下，网络宽度如何影响扰动稳定性和整体鲁棒性？
• RQ2 随着宽度增加，自然准确度、扰动稳定性与鲁棒准确度之间的关系是？
• RQ3 更大的鲁棒正则化（lambda）能否缓解由宽度引起的稳定性和鲁棒性下降？
• RQ4 是否存在一种考虑宽度的正则化策略能够在不进行大量超参数调优的情况下提升鲁棒性？

主要发现

• 更宽的网络通常自然准确度更高，但扰动稳定性更差，因此未必带来鲁棒性提升。
• 由于局部Lipschitz常数增大，扰动稳定性随宽度下降，且更宽的网络具有更大的输入梯度范数。
• 这个权衡更能在自然准确度与扰动稳定性之间描述，而非仅在自然与鲁棒准确度之间。
• 增加更宽模型的鲁棒正则化参数 lambda 可以改善扰动稳定性和鲁棒性。
• Width Adjusted Regularization（WAR）方法可以随宽度调整 lambda，节省调参时间，并在各种架构和数据集上提升鲁棒性。