[Paper Review] Unlabeled Data Improves Adversarial Robustness

Yair Carmon, Aditi Raghunathan+3 more|arXiv (Cornell University)|May 31, 2019

Adversarial Robustness in Machine Learning58 references73 citations

TL;DR

The paper shows that unlabeled data via semi-supervised self-training can substantially improve adversarial robustness, both theoretically in a Gaussian model and empirically on CIFAR-10 and SVHN, outperforming prior robust methods.

ABSTRACT

We demonstrate, theoretically and empirically, that adversarial robustness can significantly benefit from semisupervised learning. Theoretically, we revisit the simple Gaussian model of Schmidt et al. that shows a sample complexity gap between standard and robust classification. We prove that unlabeled data bridges this gap: a simple semisupervised learning procedure (self-training) achieves high robust accuracy using the same number of labels required for achieving high standard accuracy. Empirically, we augment CIFAR-10 with 500K unlabeled images sourced from 80 Million Tiny Images and use robust self-training to outperform state-of-the-art robust accuracies by over 5 points in (i) $\ell_\infty$ robustness against several strong attacks via adversarial training and (ii) certified $\ell_2$ and $\ell_\infty$ robustness via randomized smoothing. On SVHN, adding the dataset's own extra training set with the labels removed provides gains of 4 to 10 points, within 1 point of the gain from using the extra labels.

Motivation & Objective

• Motivate adversarial robustness as a stability problem that can leverage unlabeled data.
• Show a sample complexity gap between standard and robust classification in a Gaussian model and how unlabeled data bridges it.
• Propose Robust Self-Training (RST) combining pseudo-labels with robust training.
• Demonstrate empirical gains on CIFAR-10 and SVHN using large unlabeled datasets and randomized smoothing for certification.

Proposed method

• Analyze a high-dimensional Gaussian model to compare standard and robust learning with and without unlabeled data.
• Apply self-training where an intermediate model labels unlabeled data to form a final robust classifier.
• Introduce Robust Self-Training (RST) that minimizes a robust loss combining labeled and pseudo-labeled data.
• Use adversarial training (PG) or stability training (randomized smoothing) to approximate the robustness objective.
• Evaluate on CIFAR-10 with 500K unlabeled Tiny Images and on SVHN with extra unlabeled data, measuring both heuristic robustness and certified robustness.

Experimental results

Research questions

• RQ1 Does unlabeled data reduce the sample complexity gap between standard and robust classification?
• RQ2 Can self-training with pseudo-labels lead to high robust accuracy using the same number of labels as standard accuracy?
• RQ3 How much robustness gain do unlabeled data provide on CIFAR-10 and SVHN under adversarial and certified evaluations?
• RQ4 How sensitive is robust self-training to irrelevant unlabeled data and to the amount of unlabeled data?
• RQ5 How does robust self-training compare to other semi-supervised and data augmentation approaches in robustness?

Key findings

• In the Gaussian model, self-training with O(n0 ε^2 sqrt(d/n0)) unlabeled samples plus n0 labels achieves high robust accuracy, bridging the previously observed gap.
• On CIFAR-10, robust self-training with 500K unlabeled Tiny Images outperforms state-of-the-art robust accuracies by over 5–7 points in adversarial robustness and improves certified ℓ2/ℓ∞ robustness.
• RST yields a 4–10 point robustness gain on SVHN when using unlabeled extra data, with pseudo-labels contributing most of the benefit (labels offer little additional gain).
• Robust self-training can outperform other semi-supervised methods such as VAT in robustness gains; pseudo-labeling-driven unlabeled data largely drives improvements.
• Certified robustness via stability training with unlabeled data achieves strong ℓ2 robustness and competitive ℓ∞ certification, with standard accuracy gains as well.