[Paper Review] Adversarially Robust Generalization Just Requires More Unlabeled Data

Runtian Zhai, Tianle Cai+5 more|arXiv (Cornell University)|Jun 3, 2019

Adversarial Robustness in Machine Learning48 references90 citations

TL;DR

The paper proves that increasing unlabeled data can improve adversarially robust generalization and introduces a semi-supervised adversarial training method that leverages unlabeled data to boost robustness on MNIST and CIFAR-10.

ABSTRACT

Neural network robustness has recently been highlighted by the existence of adversarial examples. Many previous works show that the learned networks do not perform well on perturbed test data, and significantly more labeled data is required to achieve adversarially robust generalization. In this paper, we theoretically and empirically show that with just more unlabeled data, we can learn a model with better adversarially robust generalization. The key insight of our results is based on a risk decomposition theorem, in which the expected robust risk is separated into two parts: the stability part which measures the prediction stability in the presence of perturbations, and the accuracy part which evaluates the standard classification accuracy. As the stability part does not depend on any label information, we can optimize this part using unlabeled data. We further prove that for a specific Gaussian mixture problem, adversarially robust generalization can be almost as easy as the standard generalization in supervised learning if a sufficiently large amount of unlabeled data is provided. Inspired by the theoretical findings, we further show that a practical adversarial training algorithm that leverages unlabeled data can improve adversarial robust generalization on MNIST and Cifar-10.

Motivation & Objective

• Motivate the study of adversarial robustness and the role of unlabeled data in improving generalization.
• Decompose robust risk into a stability term ( unlabeled data dependent ) and an accuracy term ( requiring labels ).
• Provide theoretical results including a general risk bound and a Gaussian mixture example showing unlabeled data can equalize robustness with standard generalization under enough unlabeled data.
• Develop and validate a practical SSL-based adversarial training algorithm that uses labeled and unlabeled data to improve robust generalization.

Proposed method

• Present a risk decomposition: RRobust ≤ E_x sup_{x' in B(x)} I(f(x') ≠ f(x)) + RHat(f) + Rad_S(F) + 3 sqrt(log(2/δ)/(2n)).
• Show that the first term depends only on unlabeled data via P_X and can be minimized with unlabeled data.
• Prove a Gaussian mixture case where with sufficient unlabeled data, robust generalization matches standard generalization in sample complexity.
• Propose Algorithm 1 (Generalized Virtual Adversarial Training over labeled and unlabeled data) by augmenting labeled robust training with an unlabeled consistency/robustness term.
• Define L1 as robust training on labeled data and L2 as a robustness objective on unlabeled data using pseudo-labels, combined as LSSL = L1 + λ L2.
• Provide the practical SSL objective that generalizes VAT by using multiple PGD steps for stronger perturbations (k ≥ 7) to improve robust generalization.

Experimental results

Research questions

• RQ1 Can unlabeled data reduce the labeled data requirement for adversarially robust generalization?
• RQ2 How can robustness objectives be formulated to leverage unlabeled data effectively?
• RQ3 Does a Gaussian mixture model exhibit the same robustness behavior with abundant unlabeled data as with labeled data?
• RQ4 Can a practical algorithm leveraging both labeled and unlabeled data outperform standard adversarial training on real datasets?

Key findings

• A two-term bound shows robust risk is bounded by a stability term (unlabeled data dependent) plus a standard PAC-style term.
• In a Gaussian mixture setting, abundant unlabeled data can make robust generalization as easy as standard generalization in terms of sample complexity.
• A practical SSL algorithm improves robust test accuracy on MNIST and CIFAR-10 over baselines using only labeled data.
• Increasing PGD steps (k) in the attack improves robust generalization when unlabeled data are used.
• The method yields higher defense success rates and competitive robust performance compared to baselines trained on larger labeled sets.