[Paper Review] Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free

Haotao N. Wang, Tianlong Chen+4 more|arXiv (Cornell University)|Jan 1, 2020

Adversarial Robustness in Machine Learning17 citations

TL;DR

This paper proposes Once-for-All Adversarial Training (OAT), a framework that enables in-situ calibration of a single trained model to explore trade-offs between standard accuracy and robustness without retraining. By using a model-conditional training approach with dual batch normalization, OAT decouples standard and adversarial feature statistics, allowing dynamic adjustment at inference time while matching or exceeding performance of models trained from scratch for specific trade-offs.

ABSTRACT

Adversarial training and its many variants substantially improve deep network robustness, yet at the cost of compromising standard accuracy. Moreover, the training process is heavy and hence it becomes impractical to thoroughly explore the trade-off between accuracy and robustness. This paper asks this new question: how to quickly calibrate a trained model in-situ, to examine the achievable trade-offs between its standard and robust accuracies, without (re-)training it many times? Our proposed framework, Once-for-all Adversarial Training (OAT), is built on an innovative model-conditional training framework, with a controlling hyper-parameter as the input. The trained model could be adjusted among different standard and robust accuracies free at testing time. As an important knob, we exploit dual batch normalization to separate standard and adversarial feature statistics, so that they can be learned in one model without degrading performance. We further extend OAT to a Once-for-all Adversarial Training and Slimming (OATS) framework, that allows for the joint trade-off among accuracy, robustness and runtime efficiency. Experiments show that, without any re-training nor ensembling, OAT/OATS achieve similar or even superior performance compared to dedicatedly trained models at various configurations. Our codes and pretrained models are available at: this https URL.

Motivation & Objective

• To address the challenge of efficiently exploring the accuracy-robustness trade-off in deep neural networks without retraining.
• To reduce the computational burden of adversarial training by enabling dynamic model calibration post-training.
• To develop a unified framework that supports joint optimization across accuracy, robustness, and inference efficiency.
• To eliminate the need for training multiple models for different accuracy-robustness configurations.

Proposed method

• Introduce a model-conditional training framework where a single hyper-parameter controls the trade-off between standard accuracy and robustness during inference.
• Employ dual batch normalization to decouple the statistics of standard and adversarial features, enabling joint learning without performance degradation.
• Train a single model that can be dynamically adjusted at test time to achieve various points on the accuracy-robustness Pareto frontier.
• Extend OAT to OATS, integrating model pruning for joint optimization of accuracy, robustness, and inference efficiency.
• Use a single forward pass with adaptive normalization statistics to switch between different configurations at inference.
• Leverage the same model architecture and weights across all configurations, avoiding retraining or ensembling.

Experimental results

Research questions

• RQ1 Can a single trained model dynamically adjust its accuracy and robustness trade-off at inference time without retraining?
• RQ2 How effectively can dual batch normalization separate standard and adversarial feature statistics without degrading performance?
• RQ3 What is the performance of OAT compared to models trained specifically for individual accuracy-robustness configurations?
• RQ4 Can OAT be extended to jointly optimize for inference efficiency, accuracy, and robustness?
• RQ5 How does OAT's in-situ calibration compare to existing methods in terms of computational cost and model flexibility?

Key findings

• OAT enables dynamic, in-situ calibration of a single model across multiple accuracy-robustness trade-offs without any retraining or ensembling.
• The dual batch normalization mechanism successfully decouples standard and adversarial feature statistics, maintaining high performance across all configurations.
• OAT achieves performance comparable to or better than models trained from scratch for specific accuracy-robustness settings.
• The OATS extension allows joint optimization of accuracy, robustness, and inference efficiency, with minimal additional cost.
• Experiments confirm that the proposed framework achieves state-of-the-art trade-off performance across various model configurations and datasets.