[论文解读] The Dimpled Manifold Model of Adversarial Examples in Machine Learning

Adi Shamir, Odelia Melamed+1 more|arXiv (Cornell University)|Jun 18, 2021

Adversarial Robustness in Machine Learning参考文献 16被引用 4

一句话总结

本文提出了Dimpled Manifold Model（凹陷流形模型），一种解释深度学习中对抗样本的新概念框架。该模型认为，在训练过程中，决策边界首先紧密贴合低维数据流形，随后进入一个缓慢的‘凹陷’阶段，形成浅层凸起以正确分类样本。关键洞见在于，对抗样本自然源于这一凹陷过程，解释了其微小扰动及在$L_\infty$攻击中呈现噪声样外观的原因。

ABSTRACT

The extreme fragility of deep neural networks, when presented with tiny perturbations in their inputs, was independently discovered by several research groups in 2013. However, despite enormous effort, these adversarial examples remained a counterintuitive phenomenon with no simple testable explanation. In this paper, we introduce a new conceptual framework for how the decision boundary between classes evolves during training, which we call the {\em Dimpled Manifold Model}. In particular, we demonstrate that training is divided into two distinct phases. The first phase is a (typically fast) clinging process in which the initially randomly oriented decision boundary gets very close to the low dimensional image manifold, which contains all the training examples. Next, there is a (typically slow) dimpling phase which creates shallow bulges in the decision boundary that move it to the correct side of the training examples. This framework provides a simple explanation for why adversarial examples exist, why their perturbations have such tiny norms, and why they look like random noise rather than like the target class. This explanation is also used to show that a network that was adversarially trained with incorrectly labeled images might still correctly classify most test images, and to show that the main effect of adversarial training is just to deepen the generated dimples in the decision boundary. Finally, we discuss and demonstrate the very different properties of on-manifold and off-manifold adversarial perturbations. We describe the results of numerous experiments which strongly support this new model, using both low dimensional synthetic datasets and high dimensional natural datasets.

研究动机与目标

• 为深度神经网络中对抗样本的存在及其特性提供一种简单、可检验且直观的解释。
• 解决长期存在的谜题：为何对抗扰动如此微小，且常表现为随机噪声。
• 解释为何对抗样本通常位于数据流形之外，以及这与模型训练动态的关系。
• 阐明流形上与流形外对抗扰动之间的区别及其影响。
• 证明对抗训练主要加深决策边界中已有的凹陷，而非从根本上改变其结构。

提出的方法

• 提出两阶段训练过程：(1) 快速的‘贴合’阶段，决策边界迅速贴近低维数据流形；(2) 缓慢的‘凹陷’阶段，形成浅层凸起以将边界推至训练样本的正确一侧。
• 使用深度自编码器对数据流形建模，将其近似为输入空间中分段线性的连续曲面，并在每一点使用k维基底。
• 分析数据流形与决策边界之间的几何关系，表明输入空间的高维度(n)与流形的低维度(k)之间的巨大比值$\sqrt{n/k}$，解释了对抗样本的普遍性。
• 使用$L_2$和$L_\infty$范数攻击，表明$L_\infty$扰动因归一化而呈现随机噪声外观，而$L_2$扰动则因未归一化的梯度而保留物体结构。
• 采用合成2D数据集和高维自然数据集（如ImageNet）进行实证验证，将决策边界可视化为地形图。
• 分析对抗训练对凹陷深度的影响，表明其主要加深已有凹陷，而非创建新凹陷。

实验结果

研究问题

• RQ1 为何存在对抗样本，且其扰动范数为何如此之小？
• RQ2 为何$L_\infty$-范数对抗扰动呈现为随机噪声，而$L_2$-范数扰动则与原始物体相似？
• RQ3 决策边界在训练过程中如何演化，低维数据流形在此演化中起何作用？
• RQ4 流形上与流形外对抗扰动有何区别，它们与泛化和鲁棒性的关系如何？
• RQ5 为何对抗训练主要加深已有凹陷而非根本性重构决策边界？

主要发现

• 决策边界经历两阶段演化：首先快速贴合数据流形，随后缓慢凹陷，形成浅层凸起以正确分类样本。
• 对抗样本自然源于凹陷过程，解释了其存在性及扰动范数微小的原因。
• $L_\infty$-归一化梯度在FGSM和PGD攻击中导致扰动呈现为随机噪声，因为所有像素变化均被缩放至相同的最大幅度$\epsilon$，与原始梯度大小无关。
• 相比之下，$L_2$-归一化攻击保留梯度方向，导致扰动沿物体结构分布，显得不那么随机。
• 随机向量范数与其在数据流形上投影之比的期望值受$\sqrt{n/k}$限制，解释了为何对抗样本在高维空间中普遍存在。
• 对抗训练主要加深决策边界中已有的凹陷，而非创建新凹陷，这解释了其为何能提升鲁棒性而不显著改变模型结构。