[论文解读] Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free

Haotao N. Wang, Tianlong Chen+4 more|arXiv (Cornell University)|Jan 1, 2020

Adversarial Robustness in Machine Learning被引用 17

一句话总结

本文提出一次性对抗训练（OAT），一种框架，可在不重新训练的情况下，通过在推理时动态调整单一训练模型，实现标准准确率与鲁棒性之间的权衡探索。通过使用带有双重批量归一化（dual batch normalization）的模型条件训练方法，OAT 解耦了标准特征与对抗特征的统计特性，从而在推理时实现动态调整，同时性能与针对特定权衡从头训练的模型相当或更优。

ABSTRACT

Adversarial training and its many variants substantially improve deep network robustness, yet at the cost of compromising standard accuracy. Moreover, the training process is heavy and hence it becomes impractical to thoroughly explore the trade-off between accuracy and robustness. This paper asks this new question: how to quickly calibrate a trained model in-situ, to examine the achievable trade-offs between its standard and robust accuracies, without (re-)training it many times? Our proposed framework, Once-for-all Adversarial Training (OAT), is built on an innovative model-conditional training framework, with a controlling hyper-parameter as the input. The trained model could be adjusted among different standard and robust accuracies free at testing time. As an important knob, we exploit dual batch normalization to separate standard and adversarial feature statistics, so that they can be learned in one model without degrading performance. We further extend OAT to a Once-for-all Adversarial Training and Slimming (OATS) framework, that allows for the joint trade-off among accuracy, robustness and runtime efficiency. Experiments show that, without any re-training nor ensembling, OAT/OATS achieve similar or even superior performance compared to dedicatedly trained models at various configurations. Our codes and pretrained models are available at: this https URL.

研究动机与目标

• 为解决在不重新训练的情况下高效探索深度神经网络中准确率-鲁棒性权衡的挑战。
• 通过在训练后实现动态模型校准，降低对抗训练的计算负担。
• 开发一个统一框架，支持准确率、鲁棒性和推理效率的联合优化。
• 消除为不同准确率-鲁棒性配置训练多个模型的需求。

提出的方法

• 提出一种模型条件训练框架，其中单个超参数在推理时控制标准准确率与鲁棒性之间的权衡。
• 采用双重批量归一化，解耦标准特征与对抗特征的统计特性，实现在不降低性能的前提下联合学习。
• 训练一个单一模型，该模型可在测试时动态调整，以实现准确率-鲁棒性帕累托前沿上的多个点。
• 将 OAT 扩展为 OATS，集成模型剪枝以实现准确率、鲁棒性和推理效率的联合优化。
• 在推理时通过一次前向传播与自适应归一化统计，切换至不同配置。
• 在所有配置中复用相同的模型架构与权重，避免重新训练或模型集成。

实验结果

研究问题

• RQ1 一个单一训练好的模型是否能在推理时无需重新训练即可动态调整其准确率与鲁棒性权衡？
• RQ2 双重批量归一化在不降低性能的前提下，能否有效分离标准特征与对抗特征的统计特性？
• RQ3 与为特定准确率-鲁棒性配置专门训练的模型相比，OAT 的性能如何？
• RQ4 OAT 是否可扩展以实现推理效率、准确率与鲁棒性的联合优化？
• RQ5 与现有方法相比，OAT 的在推理时校准在计算成本与模型灵活性方面表现如何？

主要发现

• OAT 实现了无需任何重新训练或模型集成的、在多个准确率-鲁棒性权衡下的动态、在推理时校准。
• 双重批量归一化机制成功解耦了标准特征与对抗特征的统计特性，同时在所有配置下保持了高性能。
• OAT 的性能与为特定准确率-鲁棒性设置从头训练的模型相当或更优。
• OATS 扩展实现了准确率、鲁棒性与推理效率的联合优化，且额外开销极小。
• 实验结果证实，所提框架在多种模型配置与数据集上均实现了最先进的权衡性能。