[论文解读] HYDRA: Pruning Adversarially Robust Neural Networks

Vikash Sehwag, Shiqi Wang+2 more|arXiv (Cornell University)|Feb 24, 2020

Adversarial Robustness in Machine Learning参考文献 56被引用 31

一句话总结

HYDRA 将修剪重新表述为一个由鲁棒训练目标引导的经验风险最小化问题，从而在多种数据集和鲁棒训练方法下保持良性和对抗鲁棒性，同时实现高度压缩的网络。

ABSTRACT

In safety-critical but computationally resource-constrained applications, deep learning faces two key challenges: lack of robustness against adversarial attacks and large neural network size (often millions of parameters). While the research community has extensively explored the use of robust training and network pruning independently to address one of these challenges, only a few recent works have studied them jointly. However, these works inherit a heuristic pruning strategy that was developed for benign training, which performs poorly when integrated with robust training techniques, including adversarial training and verifiable robust training. To overcome this challenge, we propose to make pruning techniques aware of the robust training objective and let the training objective guide the search for which connections to prune. We realize this insight by formulating the pruning objective as an empirical risk minimization problem which is solved efficiently using SGD. We demonstrate that our approach, titled HYDRA, achieves compressed networks with state-of-the-art benign and robust accuracy, simultaneously. We demonstrate the success of our approach across CIFAR-10, SVHN, and ImageNet dataset with four robust training techniques: iterative adversarial training, randomized smoothing, MixTrain, and CROWN-IBP. We also demonstrate the existence of highly robust sub-networks within non-robust networks. Our code and compressed networks are publicly available at \url{https://github.com/inspire-group/compactness-robustness}.

研究动机与目标

• 在各种鲁棒训练目标下，提出一种能够保持对抗鲁棒性的修剪方法的动机。
• 开发一个使用鲁棒训练损失来引导连接性修剪的修剪框架。
• 实现高剪枝率（高稀疏度），而不牺牲良性准确性或鲁棒性。
• 在多种鲁棒性范式（对抗性训练、可验证鲁棒性、随机平滑、MixTrain 等）上展示泛化能力。

提出的方法

• 将修剪建模为带有鲁棒性感知损失的经验风险最小化问题。
• 使用基于重要性分数的优化来搜索应修剪的连接。
• 引入与预训练权重成正比的重要性分数的放缩初始化，以提高修剪性能。
• 将修剪与若干鲁棒训练目标整合（对抗性训练、通过 MixTrain 和 CROWN-IBP 的可验证鲁棒性、随机平滑）。
• 修剪后仅对未被修剪的连接进行微调以恢复性能。

实验结果

研究问题

• RQ1 修剪是否可以考虑到鲁棒训练目标，而不是依赖于良性训练的启发式方法？
• RQ2 鲁棒性感知的修剪是否能够在不牺牲良性和鲁棒性准确性的前提下实现高压缩，适用于多种鲁棒性目标？
• RQ3 在非鲁棒网络中是否可以发现鲁棒子网络，修剪是否能在不同鲁棒性目标下揭示它们？
• RQ4 HYDRA 在数据集和架构上如何与现有鲁棒修剪方法（Adv-LWM、Adv-ADMM）进行比较？
• RQ5 该方法是否可扩展到大型数据集（如 ImageNet）以及不同的鲁棒性表达（对抗性训练、可验证鲁棒性、平滑、MixTrain 等）？
• RQ6

主要发现

• HYDRA 在 CIFAR-10、SVHN 和 ImageNet 上与先前的修剪方法相比，在良性准确性和鲁棒性方面均有显著提升。
• 重要性分数的放缩初始化是性能的关键驱动因素，能够实现更快收敛和更好的修剪结果。
• HYDRA 在多种架构和修剪比率下优于 Adv-LWM 与 Adv-ADMM，包括非常高的修剪率（高达 99%）。
• 该方法可泛化到四种鲁棒训练目标：迭代对抗性训练、随机平滑、MixTrain 和 CROWN-IBP，在 era 和 vra 指标中观察到鲁棒提升。
• 在非鲁棒网络中存在高度鲁棒的子网络，表明即使在预训练阶段并非完全鲁棒也存在鲁棒潜力。
• 修剪后的网络在更强攻击下可以维持甚至超过预训练的鲁棒性，并且可以进一步量化（8 位）而损失极小。