[Paper Review] Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty

Dan Hendrycks, Mantas Mazeika+2 more|arXiv (Cornell University)|Jun 28, 2019

Adversarial Robustness in Machine LearningComputer Science39 references343 citations

TL;DR

The paper demonstrates that self-supervised rotation prediction can improve robustness to adversarial perturbations, label noise, and common corruptions, and greatly enhance out-of-distribution detection, even surpassing fully supervised methods in some cases.

ABSTRACT

Self-supervision provides effective representations for downstream tasks without requiring labels. However, existing approaches lag behind fully supervised training and are often not thought beneficial beyond obviating or reducing the need for annotations. We find that self-supervision can benefit robustness in a variety of ways, including robustness to adversarial examples, label corruption, and common input corruptions. Additionally, self-supervision greatly benefits out-of-distribution detection on difficult, near-distribution outliers, so much so that it exceeds the performance of fully supervised methods. These results demonstrate the promise of self-supervision for improving robustness and uncertainty estimation and establish these tasks as new axes of evaluation for future self-supervised learning research.

Motivation & Objective

• Motivate robust and reliable models when labeled data are scarce or unlabeled signals are valuable.
• Investigate whether self-supervised objectives can enhance robustness to adversarial examples, label corruption, and input corruptions.
• Examine the impact of self-supervision on out-of-distribution detection under challenging near-distribution conditions.

Proposed method

• Train a classifier with an auxiliary head that predicts image rotations (0, 90, 180, 270 degrees) using a self-supervised loss.
• Combine standard supervised cross-entropy loss with a rotation-based self-supervised loss, weighted by lambda (set to 0.5).
• Modify the PGD adversarial training objective to maximize both classification and rotation losses during training.
• Evaluate robustness to adversarial perturbations (PGD 20-step and 100-step) and common corruptions (CIFAR-10-C), using WRN-40-2 networks.
• Test OOD detection by augmenting supervised representations with rotation-based self-supervision and using a KL-divergence plus rotation-score detector.
• Extend experiments to ImageNet with larger self-supervised objectives and architectural variations (e.g., self-attention) to improve OOD performance.

Experimental results

Research questions

• RQ1 Can self-supervised learning improve robustness to adversarial perturbations beyond standard adversarial training?
• RQ2 Does rotation-based self-supervision improve robustness to common input corruptions and label noise?
• RQ3 Can self-supervision enhance out-of-distribution detection, particularly for near-distribution OOD cases?
• RQ4 Do self-supervised techniques scale to larger datasets like ImageNet and yield gains over fully supervised models?

Key findings

• Auxiliary rotation self-supervision yields 5.6 percentage point improvements over standard PGD training under both 20-step and 100-step adversaries.
• The rotation-based approach withstands up to 11% larger perturbations than PGD training without loss in performance.
• When combined with TRADES, rotations provide a 1.22 percentage point boost over TRADES and a 7.79 percentage point boost over PGD training in robust accuracy.
• On CIFAR-10-C, rotation self-supervision increases average robustness from 72.3% to 76.9% (4.6 percentage points) with clean accuracy remaining near constant.
• For out-of-distribution detection, rotation-based self-supervision improves AUROC over the baseline maximum softmax probability method by an average of 4.8 percentage points on CIFAR-10 experiments.
• On CIFAR-10 one-class OOD tasks, self-supervised RotNet-based detectors surpass several fully supervised and other self-supervised baselines, including outperforming some methods that use outlier data.