[Paper Review] Adversarial Distributional Training for Robust Deep Learning

Yinpeng Dong, Zhijie Deng+3 more|arXiv (Cornell University)|Feb 14, 2020

Adversarial Robustness in Machine Learning85 references34 citations

TL;DR

Presents Adversarial Distributional Training (ADT), a minimax framework that learns adversarial distributions around inputs to improve robustness against unseen attacks, with three parameterizations and empirical tests on CIFAR-10/100 and SVHN.

ABSTRACT

Adversarial training (AT) is among the most effective techniques to improve model robustness by augmenting training data with adversarial examples. However, most existing AT methods adopt a specific attack to craft adversarial examples, leading to the unreliable robustness against other unseen attacks. Besides, a single attack algorithm could be insufficient to explore the space of perturbations. In this paper, we introduce adversarial distributional training (ADT), a novel framework for learning robust models. ADT is formulated as a minimax optimization problem, where the inner maximization aims to learn an adversarial distribution to characterize the potential adversarial examples around a natural one under an entropic regularizer, and the outer minimization aims to train robust models by minimizing the expected loss over the worst-case adversarial distributions. Through a theoretical analysis, we develop a general algorithm for solving ADT, and present three approaches for parameterizing the adversarial distributions, ranging from the typical Gaussian distributions to the flexible implicit ones. Empirical results on several benchmarks validate the effectiveness of ADT compared with the state-of-the-art AT methods.

Motivation & Objective

• Motivate robustness against unseen adversarial attacks beyond single-attack adversarial training.
• Introduce a distributional minimax formulation that treats perturbations as distributions rather than points.
• Regularize to prevent collapse to Delta distributions and encourage diverse adversarial examples.
• Provide three practical instantiations of adversarial distributions and analyze their impact on robustness.

Proposed method

• Formulate ADT as a minimax problem: min_theta (1/n) sum_i max_p(delta_i) E_{p}[L(f_theta(x_i+delta_i), y_i)].
• Add an entropic regularizer to the inner objective to prevent degeneracy into Delta distributions: J(p, theta) = E_p[L(...)] + lambda H(p).
• Parameterize adversarial distributions with three approaches: (i) ADT EXP using explicit Gaussian-based transforms delta = epsilon*tanh(u), u~N(mu, diag(sigma^2)); (ii) ADT EXP-AM amortized with a generator g_phi conditioned on x; (iii) ADT IMP-AM implicit distributions via a generator with latent z and implicit density via variational entropy estimates.
• Provide a general algorithm (Alg. 1) based on Danskin-like sequential optimization: solve inner max to get p*, then update theta via gradient of J at p*.],
• research_questions=[
• 1) Can learning distributions over adversarial perturbations improve robustness to a wider range of attacks compared to point-wise adversarial training?
• 2) How does entropic regularization affect the diversity and effectiveness of the learned adversarial perturbations?
• 3) How do explicit, amortized-explicit, and implicit adversarial distribution parameterizations compare in terms of robustness and training efficiency?
• 4) Do ADT methods maintain robustness under white-box and black-box attacks across standard benchmarks?

Experimental results

Research questions

• RQ1

Key findings

• ADT-based methods consistently improve robustness against a suite of white-box attacks (FGSM, PGD variants, MIM, C&W, FeaAttack) compared with standard training and competitive adversarial trainings.
• On CIFAR-10/10 under white-box attacks, ADT variants achieve higher attack-robust accuracy than many baselines, with ADT EXP often outperforming EXP-AM and IMP-AM in several settings.
• ADT-based methods demonstrate better robustness under black-box transfers and SPSA queries, indicating reduced gradient masking and authentic robustness gains.
• The entropy regularization enables adversarial distributions to cover a broader perturbation space, yielding a smoother loss landscape around natural inputs.
• AMORTIZED variants (EXP-AM, IMP-AM) offer faster training with comparable robustness, though explicit EXP may yield marginally stronger defenses in some cases.
• Empirical results on CIFAR-10, CIFAR-100, and SVHN validate the effectiveness of ADT against state-of-the-art AT methods.