[Paper Review] Efficient Adversarial Training with Transferable Adversarial Examples

Haizhong Zheng, Ziqi Zhang+3 more|arXiv (Cornell University)|Dec 27, 2019

Adversarial Robustness in Machine Learning41 references9 citations

TL;DR

This paper proposes ATTA (Adversarial Training with Transferable Adversarial Examples), a novel method that reuses adversarial perturbations across training epochs by leveraging high transferability between models from neighboring epochs. By accumulating attack strength through epoch-wise reuse, ATTA achieves comparable or better robustness than PGD-based adversarial training while reducing training time by up to 14.1× on CIFAR10 and improving adversarial accuracy by up to 7.2%.

ABSTRACT

Adversarial training is an effective defense method to protect classification models against adversarial attacks. However, one limitation of this approach is that it can require orders of magnitude additional training time due to high cost of generating strong adversarial examples during training. In this paper, we first show that there is high transferability between models from neighboring epochs in the same training process, i.e., adversarial examples from one epoch continue to be adversarial in subsequent epochs. Leveraging this property, we propose a novel method, Adversarial Training with Transferable Adversarial Examples (ATTA), that can enhance the robustness of trained models and greatly improve the training efficiency by accumulating adversarial perturbations through epochs. Compared to state-of-the-art adversarial training methods, ATTA enhances adversarial accuracy by up to 7.2% on CIFAR10 and requires 12~14x less training time on MNIST and CIFAR10 datasets with comparable model robustness.

Motivation & Objective

• To address the high computational cost of generating strong adversarial examples during adversarial training.
• To investigate whether adversarial perturbations from one training epoch can remain effective in subsequent epochs.
• To develop a method that reuses adversarial perturbations across epochs to reduce training time without sacrificing model robustness.
• To improve the efficiency and effectiveness of iterative attack-based adversarial training methods.
• To demonstrate the practicality of the approach on large-scale datasets like ImageNet.

Proposed method

• The method reuses adversarial perturbations generated in one epoch for adversarial training in the next, exploiting high transferability between models from neighboring epochs.
• It applies a single attack iteration per epoch to generate adversarial examples, reusing the perturbation from the previous epoch as the starting point.
• The approach uses a connection function to handle distribution shifts between epochs, ensuring perturbations remain effective across model updates.
• It integrates with existing adversarial training frameworks such as Madry’s method (MAT) and TRADES, enhancing them without modifying their core architecture.
• Inverse data augmentation (i.d.a.) is applied to further improve transferability between epochs.
• The attack loss function is carefully selected—Equation 3 is preferred over Equation 2 for better stability and transferability in the context of epoch-wise reuse.

Experimental results

Research questions

• RQ1 Is there high transferability between models trained at consecutive epochs during adversarial training?
• RQ2 Can adversarial perturbations from one epoch be effectively reused in subsequent epochs to reduce attack iterations?
• RQ3 Does reusing perturbations across epochs lead to improved training efficiency without sacrificing model robustness?
• RQ4 How does the choice of attack loss function affect transferability and performance in epoch-wise adversarial training?
• RQ5 Can the proposed method scale to large-scale datasets like ImageNet with minimal computational overhead?

Key findings

• ATTA reduces training time by up to 14.1× on CIFAR10 and 12.2× on MNIST compared to traditional PGD-based adversarial training.
• With only one attack iteration per epoch, ATTA-1 achieves adversarial accuracy of 96.31% on MNIST in 5 minutes, compared to 65 minutes with MAT.
• On CIFAR10, ATTA achieves comparable adversarial accuracy to MAT in about 2 hours, compared to over 33 hours with MAT.
• ATTA enhances adversarial accuracy by up to 7.2% over MAT on CIFAR10, demonstrating improved robustness.
• On ImageNet, ATTA achieves comparable robustness in 97 hours using a single 2080 Ti GPU, showing scalability.
• Inverse data augmentation increases transferability, improving robustness by approximately 6% in ablation studies.