[论文解读] MixTrain: Scalable Training of Formally Robust Neural Networks.

Shiqi Wang, Yizheng Chen+2 more|arXiv (Cornell University)|Nov 6, 2018

Adversarial Robustness in Machine Learning参考文献 56被引用 74

一句话总结

MixTrain 引入了两种高效技术——随机鲁棒近似和动态混合训练，以在计算开销最小的情况下，实现可验证鲁棒神经网络训练的可扩展性。通过使用随机数据子采样和自适应损失平衡，它在 ImageNet-200 上实现了高达 95.2% 的可验证鲁棒准确率，相较于最先进的可验证鲁棒方法和对抗鲁棒方法，训练速度分别提升了 15 倍和 3 倍。

ABSTRACT

Making neural networks robust against adversarial inputs has resulted in an arms race between new defenses and attacks. The most promising defenses, adversarially robust training and verifiably robust training, have limitations that restrict their practical applications. The adversarially robust training only makes the networks robust against a subclass of attackers and we reveal such weaknesses by developing a new attack based on interval gradients. By contrast, verifiably robust training provides protection against any L-p norm-bounded attacker but incurs orders of magnitude more computational and memory overhead than adversarially robust training. We propose two novel techniques, stochastic robust approximation and dynamic mixed training, to drastically improve the efficiency of verifiably robust training without sacrificing verified robustness. We leverage two critical insights: (1) instead of over the entire training set, sound over-approximations over randomly subsampled training data points are sufficient for efficiently guiding the robust training process; and (2) We observe that the test accuracy and verifiable robustness often conflict after certain training epochs. Therefore, we use a dynamic loss function to adaptively balance them for each epoch. We designed and implemented our techniques as part of MixTrain and evaluated it on six networks trained on three popular datasets including MNIST, CIFAR, and ImageNet-200. Our evaluations show that MixTrain can achieve up to $95.2\%$ verified robust accuracy against $L_\infty$ norm-bounded attackers while taking $15$ and $3$ times less training time than state-of-the-art verifiably robust training and adversarially robust training schemes, respectively. Furthermore, MixTrain easily scales to larger networks like the one trained on ImageNet-200, significantly outperforming the existing verifiably robust training methods.

研究动机与目标

• 为解决可验证鲁棒训练的高计算和内存开销问题，尽管其具有强大的鲁棒性保证，但该问题限制了其实际应用。
• 克服对抗鲁棒训练对新型攻击（尤其是基于区间梯度的攻击）的脆弱性。
• 实现在大规模数据集（如 ImageNet-200）上高效训练可验证鲁棒网络，而此前的方法因可扩展性问题而失效。
• 通过动态调整损失函数，在训练过程中平衡测试准确率与可验证鲁棒性。

提出的方法

• 随机鲁棒近似用对随机子采样训练点的过近似替代全批量过近似，降低计算成本，同时保持鲁棒性保证。
• 动态混合训练引入一种自适应损失函数，根据各训练轮次的性能冲突，动态平衡测试准确率与可验证鲁棒性。
• 该方法利用区间梯度识别对抗鲁棒模型中的弱点，从而指导设计更具鲁棒性的训练策略。
• 通过基于区间的网络输出边界保持可验证鲁棒性，确保对所有 L∞-范数有界的对抗扰动提供形式化保证。
• 训练流程将这些技术整合到统一框架中，实现端到端训练，且对现有可验证鲁棒训练流程的修改极少。

实验结果

研究问题

• RQ1 可验证鲁棒训练能否在不牺牲鲁棒性保证的前提下，实现对 ImageNet-200 等大规模模型和数据集的可扩展性？
• RQ2 通过子采样训练数据进行过近似，能否在不损害鲁棒性的前提下降低可验证鲁棒训练的计算开销？
• RQ3 在训练过程中动态调整损失函数，能否改善测试准确率与可验证鲁棒性之间的权衡？
• RQ4 与最先进的对抗鲁棒和可验证鲁棒训练方法相比，所提出方法在效率和鲁棒性方面表现如何？

主要发现

• MixTrain 在 ImageNet-200 上对 L∞-范数有界的对抗攻击实现了高达 95.2% 的可验证鲁棒准确率，显著优于先前的可验证鲁棒训练方法。
• MixTrain 的训练时间比最先进的可验证鲁棒训练方法快 15 倍，使其在大规模数据集上更具实用性。
• 与对抗鲁棒训练相比，MixTrain 将训练时间减少了 3 倍，同时保持了更强的鲁棒性保证。
• 该方法成功扩展到更大的网络，包括在 ImageNet-200 上训练的网络，而此前的可验证鲁棒训练方法因计算资源限制而失效。
• 动态混合训练通过按训练轮次自适应平衡损失函数，有效解决了测试准确率与可验证鲁棒性之间的冲突。