[论文解读] The curse of overparametrization in adversarial training: Precise analysis of robust generalization for random features regression

Hamed Hassani, Adel Javanmard|arXiv (Cornell University)|Jan 13, 2022

Adversarial Robustness in Machine Learning被引用 4

一句话总结

本文在高维渐近框架下，对对抗训练的随机特征模型中的鲁棒泛化进行了精确的理论分析。研究发现，虽然过参数化有利于标准泛化，但会因对对抗扰动的敏感性增加而恶化鲁棒泛化误差，揭示了模型设计中的根本性权衡。

ABSTRACT

Successful deep learning models often involve training neural network architectures that contain more parameters than the number of training samples. Such overparametrized models have been extensively studied in recent years, and the virtues of overparametrization have been established from both the statistical perspective, via the double-descent phenomenon, and the computational perspective via the structural properties of the optimization landscape. Despite the remarkable success of deep learning architectures in the overparametrized regime, it is also well known that these models are highly vulnerable to small adversarial perturbations in their inputs. Even when adversarially trained, their performance on perturbed inputs (robust generalization) is considerably worse than their best attainable performance on benign inputs (standard generalization). It is thus imperative to understand how overparametrization fundamentally affects robustness. In this paper, we will provide a precise characterization of the role of overparametrization on robustness by focusing on random features regression models (two-layer neural networks with random first layer weights). We consider a regime where the sample size, the input dimension and the number of parameters grow in proportion to each other, and derive an asymptotically exact formula for the robust generalization error when the model is adversarially trained. Our developed theory reveals the nontrivial effect of overparametrization on robustness and indicates that for adversarially trained random features models, high overparametrization can hurt robust generalization.

研究动机与目标

• 理解过参数化对对抗训练模型中鲁棒泛化的基本影响。
• 分析在样本量、输入维度和模型参数成比例增长条件下，随机特征回归模型中的鲁棒泛化误差。
• 在高维极限下，推导鲁棒泛化误差的渐近精确公式。
• 揭示过参数化在对抗鲁棒性中非单调且可能有害的作用。

提出的方法

• 分析采用高维渐近框架，其中样本量、输入维度和参数数量成比例增长。
• 模型采用具有随机第一层权重的两层神经网络（随机特征回归）。
• 通过涉及Moreau包络和高斯等价性质的变分公式推导鲁棒泛化误差。
• 方法利用随机矩阵理论工具，包括Marchenko-Pastur分布的Stieltjes变换以及核矩阵的谱近似。
• 分析证明了极限目标函数的严格凸性，确保最优解的唯一性。
• 推导过程采用变量变换和大数集中律，以分析优化景观的渐近行为。

(a) $\varepsilon=10^{-7}$

实验结果

研究问题

• RQ1 过参数化如何影响对抗训练的随机特征模型中的鲁棒泛化误差？
• RQ2 在过参数化模型中，标准泛化与鲁棒泛化之间是否存在根本性权衡？
• RQ3 我们能否在高维极限下推导出鲁棒泛化误差的渐近精确公式？
• RQ4 尽管标准泛化性能提升，高过参数化是否反而导致鲁棒性下降？

主要发现

• 过参数化可能恶化鲁棒泛化误差，与‘参数越多性能越好’的普遍认知相悖。
• 在模型维度成比例增长的条件下，鲁棒泛化误差被推导为渐近精确公式。
• 分析揭示了过参数化对鲁棒性的非平凡、非单调影响，表明过度复杂性会损害鲁棒性。
• 极限优化目标函数是严格凸的，确保了鲁棒和标准泛化参数的唯一最小值。
• 推导出的公式表明，鲁棒误差高于标准误差，且该差距随过参数化程度增加而扩大。
• 结果与经验观察一致：尽管容量高，对抗训练模型仍表现出脆弱性。

(b) $\varepsilon=0.1$