[Paper Review] Are Labels Required for Improving Adversarial Robustness?

Jonathan Uesato, Jean-Baptiste Alayrac+4 more|arXiv (Cornell University)|May 31, 2019

Adversarial Robustness in Machine Learning55 references92 citations

TL;DR

The paper shows unlabeled data can match much of the robustness gains of labeled data in adversarial training, introducing Unsupervised Adversarial Training (UAT) and demonstrating near-supervised performance with far fewer labels, plus improvements using uncurated data.

ABSTRACT

Recent work has uncovered the interesting (and somewhat surprising) finding that training models to be invariant to adversarial perturbations requires substantially larger datasets than those required for standard classification. This result is a key hurdle in the deployment of robust machine learning models in many real world applications where labeled data is expensive. Our main insight is that unlabeled data can be a competitive alternative to labeled data for training adversarially robust models. Theoretically, we show that in a simple statistical setting, the sample complexity for learning an adversarially robust model from unlabeled data matches the fully supervised case up to constant factors. On standard datasets like CIFAR-10, a simple Unsupervised Adversarial Training (UAT) approach using unlabeled data improves robust accuracy by 21.7% over using 4K supervised examples alone, and captures over 95% of the improvement from the same number of labeled examples. Finally, we report an improvement of 4% over the previous state-of-the-art on CIFAR-10 against the strongest known attack by using additional unlabeled data from the uncurated 80 Million Tiny Images dataset. This demonstrates that our finding extends as well to the more realistic case where unlabeled data is also uncurated, therefore opening a new avenue for improving adversarial training.

Motivation & Objective

• Motivate the use of unlabeled data to improve adversarial robustness and reduce reliance on labeled data.
• Introduce Unsupervised Adversarial Training (UAT) strategies to leverage unlabeled data for robust classifiers.
• Theoretically compare sample complexity with and without labels in a Gaussian model.
• Empirically evaluate UAT on CIFAR-10 and SVHN under strong adversarial attacks and with uncurated data.
• Push toward state-of-the-art robustness using large-scale unlabeled data.

Proposed method

• Define adversarial and natural risks and surrogate adversarial risk using an inner maximization (PGD-based) adversary.
• Propose two UAT strategies: Unsupervised Adversarial Training with Online Targets (UAT-OT) and Unsupervised Adversarial Training with Fixed Targets (UAT-FT), plus a combined variant UAT++.
• Combine supervised losses with unsupervised smoothness losses, controlled by a weighting hyperparameter λ.
• Provide a Gaussian-model theoretical analysis showing unlabeled data can match supervised sample complexity for fixed robust accuracy.
• Evaluate on CIFAR-10 and SVHN with ε=8/255 and ε=0.01, including experiments with 80 Million Tiny Images as uncurated unlabeled data.

Experimental results

Research questions

• RQ1 Can unlabeled data achieve adversarial robustness comparable to labeled data for a fixed data budget?
• RQ2 How do UAT variants (OT, FT, and ++ ) perform under standard and strong adversarial attacks on CIFAR-10 and SVHN?
• RQ3 Does uncurated unlabeled data (e.g., 80 Million Tiny Images) improve state-of-the-art robustness on CIFAR-10?
• RQ4 What is the robustness of UAT to distribution shifts between labeled and unlabeled data?
• RQ5 How robust is UAT to label noise in the pseudo-labeling process?

Key findings

• Unlabeled data can be competitive with labeled data for adversarial robustness; UAT variants improve robustness substantially over baselines using the same labeled data.
• UAT-FT and UAT++ often outperform UAT-OT, especially with larger unlabeled sets, and UAT++ nears the supervised oracle in CIFAR-10 and SVHN under FGSM threat models.
• On CIFAR-10 with 32K unlabeled samples, UAT++ achieves 54.1% robust accuracy vs 55.5% with full supervision, i.e., within 1.4 percentage points of the supervised oracle.
• With 60K unlabeled samples, UAT++ reaches 84.4% robust accuracy on SVHN, within 1.8 percentage points of the supervised oracle.
• Using uncurated data (80m Tiny Images) with larger models yields state-of-the-art robustness on CIFAR-10 against strong attacks, e.g., WRN-106 achieving 56.30% under MultiTargeted attack.
• UAT++ with 200K unlabeled 80m data and WRN-34/106 shows substantial gains over TRADES and other baselines under FGSM 20 and MultiTargeted attacks.
• The approach demonstrates robustness to label noise, with significant gains even when pseudo-labels are noisy or partially corrupted.