[Paper Review] WAT: Improve the Worst-class Robustness in Adversarial Training

Boqi Li, Weiwei Liu|arXiv (Cornell University)|Feb 8, 2023

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes WAT, a novel worst-class adversarial training framework that leverages no-regret dynamics to optimize robustness for the most vulnerable class, achieving significantly improved worst-class robust accuracy while sacrificing minimal average robust accuracy. The method outperforms state-of-the-art baselines on CIFAR-10 and ImageNet-100 across multiple attacks, with a new metric ($\rho$) demonstrating superior trade-off balance between average and worst-class performance.

ABSTRACT

Deep Neural Networks (DNN) have been shown to be vulnerable to adversarial examples. Adversarial training (AT) is a popular and effective strategy to defend against adversarial attacks. Recent works (Benz et al., 2020; Xu et al., 2021; Tian et al., 2021) have shown that a robust model well-trained by AT exhibits a remarkable robustness disparity among classes, and propose various methods to obtain consistent robust accuracy across classes. Unfortunately, these methods sacrifice a good deal of the average robust accuracy. Accordingly, this paper proposes a novel framework of worst-class adversarial training and leverages no-regret dynamics to solve this problem. Our goal is to obtain a classifier with great performance on worst-class and sacrifice just a little average robust accuracy at the same time. We then rigorously analyze the theoretical properties of our proposed algorithm, and the generalization error bound in terms of the worst-class robust risk. Furthermore, we propose a measurement to evaluate the proposed method in terms of both the average and worst-class accuracies. Experiments on various datasets and networks show that our proposed method outperforms the state-of-the-art approaches.

Motivation & Objective

• To address the robustness disparity in adversarial training, where some classes remain highly vulnerable despite strong average robustness.
• To develop a method that enhances worst-class robust accuracy without significantly degrading average robust accuracy.
• To provide a theoretical guarantee on worst-class robust risk using Rademacher complexity.
• To propose a new evaluation metric ($\rho$) that balances average and worst-class robust accuracy.
• To empirically validate the method's superiority across diverse datasets and network architectures.

Proposed method

• Proposes a min-max learning paradigm that explicitly optimizes the worst-class robust risk during adversarial training.
• Employs no-regret dynamics to solve the min-max optimization problem, ensuring convergence and stability.
• Introduces a hyperparameter $\eta$ to control the trade-off between worst-class and average robust accuracy.
• Uses a modified adversarial training objective that emphasizes the loss of the most vulnerable class.
• Applies a generalization error bound analysis based on Rademacher complexity to justify theoretical robustness.
• Proposes a new evaluation metric $\rho$ that quantifies the balance between average and worst-class robust accuracy.

Experimental results

Research questions

• RQ1 Can adversarial training be restructured to prioritize the worst-performing class without sacrificing overall robustness?
• RQ2 How can a stable and convergent optimization method be designed to minimize worst-class robust risk?
• RQ3 What is the theoretical generalization error bound for worst-class robust risk in adversarial training?
• RQ4 How can a balanced evaluation metric be defined that reflects both average and worst-class robust accuracy?
• RQ5 Does the proposed method outperform existing approaches in both worst-class and average robustness across diverse datasets and models?

Key findings

• On CIFAR-10 with ResNet-18, WAT achieves a worst-class robust accuracy of 69.5% under PGD-100 and 33.3% under CW attack, outperforming TRADES and other baselines.
• With $\eta = 0.1$, WAT achieves the highest $\rho_{cw}$ value of 0.326, indicating the best balance between average and worst-class robust accuracy.
• On WideResNet-34-10, WAT achieves the highest worst-class robust accuracy across all three attacks (PGD-100, CW, AutoAttack), with at least 1.3% improvement over baselines.
• The proposed $\rho$ metric outperforms $CV_{cw}$ as a balanced evaluation measure, as it accounts for both average and worst-class performance.
• Class-wise robust accuracy analysis confirms that WAT significantly improves the most vulnerable classes (e.g., class-4, class-5) compared to TRADES and CSL.
• Parameter analysis shows that $\eta = 0.1$ yields the best $\rho_{nat}$ and $\rho_{cw}$, while $\eta = 0.5$ achieves the highest $\rho_{pgd}$ and $\rho_{AA}$, indicating sensitivity to attack type.