[Paper Review] Amata: An Annealing Mechanism for Adversarial Training Acceleration

Nanyang Ye, Qianxiao Li+2 more|arXiv (Cornell University)|Dec 15, 2020

Adversarial Robustness in Machine Learning37 references4 citations

TL;DR

Amata proposes an annealing mechanism that dynamically reduces the number of adversarial attack iterations during training, significantly accelerating adversarial training while maintaining or improving robustness. It achieves 1/3 to 1/2 the computational time of standard methods and is compatible with existing acceleration techniques like YOPO, Free, Fast, and ATTA.

ABSTRACT

Despite the empirical success in various domains, it has been revealed that deep neural networks are vulnerable to maliciously perturbed input data that much degrade their performance. This is known as adversarial attacks. To counter adversarial attacks, adversarial training formulated as a form of robust optimization has been demonstrated to be effective. However, conducting adversarial training brings much computational overhead compared with standard training. In order to reduce the computational cost, we propose an annealing mechanism, Amata, to reduce the overhead associated with adversarial training. The proposed Amata is provably convergent, well-motivated from the lens of optimal control theory and can be combined with existing acceleration methods to further enhance performance. It is demonstrated that on standard datasets, Amata can achieve similar or better robustness with around 1/3 to 1/2 the computational time compared with traditional methods. In addition, Amata can be incorporated into other adversarial training acceleration algorithms (e.g. YOPO, Free, Fast, and ATTA), which leads to further reduction in computational time on large-scale problems.

Motivation & Objective

• Address the high computational cost of adversarial training, which stems from repeatedly solving the inner maximization problem via PGD.
• Overcome the inefficiency of using full PGD steps throughout training, especially in early stages where strong adversarial examples are less critical.
• Develop a theoretically grounded, provably convergent mechanism that reduces redundant computation in adversarial training.
• Enable seamless integration with existing adversarial training acceleration methods to further enhance performance and efficiency.
• Provide a control-theoretic foundation for adaptive training schedules in adversarial robustness.

Proposed method

• Introduce an annealing schedule that gradually reduces the number of PGD steps used in the inner maximization problem as training progresses.
• Formulate adversarial training as an optimal control problem, deriving the annealing strategy from Pontryagin’s maximum principle to balance robustness and efficiency.
• Use a time-varying step count for PGD attacks: start with fewer steps (e.g., 1–2) and increase to full steps (e.g., 10) as training advances.
• Ensure convergence by proving the method is provably convergent under standard assumptions of non-convex optimization.
• Integrate Amata with existing acceleration methods (e.g., YOPO, Free, Fast, ATTA) by modifying their training schedules without altering core algorithms.
• Leverage the loss landscape to guide the annealing schedule, ensuring that early training uses weaker adversarial examples while stronger ones are used later.

Experimental results

Research questions

• RQ1 Can reducing the number of PGD steps in adversarial training early in training significantly reduce computational cost without sacrificing robustness?
• RQ2 Is there a theoretically grounded, optimal way to schedule the number of PGD steps during adversarial training using control theory?
• RQ3 Can the annealing mechanism be combined with existing adversarial training acceleration methods to yield further performance improvements?
• RQ4 Does the proposed annealing schedule lead to provably convergent training dynamics in the context of min-max optimization?
• RQ5 How does the annealing mechanism compare to fixed-step PGD in terms of robustness and training efficiency across different datasets and architectures?

Key findings

• Amata reduces adversarial training time by approximately 1/3 to 1/2 compared to standard PGD-based methods on CIFAR-10 and ImageNet.
• On CIFAR-10, Amata+YOPO achieves 94% adversarial accuracy in 294 seconds, which is about half the time of YOPO and with slightly better robustness.
• On ImageNet, Amata+Free reduces training time from 1318 to 948 minutes to reach 30% adversarial accuracy, saving around 1/3 of the computational cost.
• Amata+ATTA achieves 58% PGD-20 accuracy 1.5 times faster than ATTA by reducing the reset frequency and step count in early epochs.
• Amata+Fast achieves 72% PGD-20 accuracy on CIFAR-10 in half the time of standard Fast adversarial training by using a weaker initialization phase.
• The method maintains or improves robustness across all benchmarks, demonstrating that early-stage weaker attacks are sufficient for effective generalization.