[Paper Review] Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning

Takeru Miyato, Shin‐ichi Maeda+2 more|arXiv (Cornell University)|Apr 13, 2017

Adversarial Robustness in Machine Learning25 references218 citations

TL;DR

This paper introduces Virtual Adversarial Training (VAT), a regularization technique that smooths the model's output distribution around inputs by perturbing them in virtual adversarial directions, enabling effective supervised and semi-supervised learning with few hyperparameters.

ABSTRACT

We propose a new regularization method based on virtual adversarial loss: a new measure of local smoothness of the conditional label distribution given input. Virtual adversarial loss is defined as the robustness of the conditional label distribution around each input data point against local perturbation. Unlike adversarial training, our method defines the adversarial direction without label information and is hence applicable to semi-supervised learning. Because the directions in which we smooth the model are only "virtually" adversarial, we call our method virtual adversarial training (VAT). The computational cost of VAT is relatively low. For neural networks, the approximated gradient of virtual adversarial loss can be computed with no more than two pairs of forward- and back-propagations. In our experiments, we applied VAT to supervised and semi-supervised learning tasks on multiple benchmark datasets. With a simple enhancement of the algorithm based on the entropy minimization principle, our VAT achieves state-of-the-art performance for semi-supervised learning tasks on SVHN and CIFAR-10.

Motivation & Objective

• Introduce a regularization concept targeting local distributional smoothness of p(y|x) without requiring labels.
• Extend regularization to semi-supervised learning by using virtual adversarial directions defined on unlabeled data.
• Provide an efficient algorithm suitable for neural networks with low computational overhead.
• Demonstrate state-of-the-art or competitive results on standard benchmarks (e.g., MNIST, CIFAR-10, SVHN) and analyze hyperparameter effects.

Proposed method

• Define local distributional smoothness (LDS) as a divergence-based measure of how much p(y|x) changes under a perturbation in the input.
• Introduce virtual adversarial perturbations r_vadv that maximize the divergence between p(y|x, theta) and p(y|x+r, theta) without requiring true labels.
• Use the current model output p(y|x, theta) as a stand-in for the unknown label distribution q(y|x) in unlabeled data (virtual labels).
• Approximate r_vadv efficiently via a power-iteration-like method that uses backpropagation to compute gradients with respect to input.
• Form the VAT objective by combining the standard negative log-likelihood with a regularizer that averages LDS over all inputs, controlled by a small set of hyperparameters (epsilon and alpha).
• Explain differences and advantages over Random Perturbation Training (RPT) and classic adversarial training, emphasizing parameterization invariance and spectral focus on the Hessian’s dominant direction.

Experimental results

Research questions

• RQ1 Can a regularization based on local distributional smoothness improve generalization in supervised learning?
• RQ2 Can the same regularization be effectively applied to semi-supervised learning using unlabeled data?
• RQ3 Does focusing smoothing in the virtual adversarial direction (vs. isotropic perturbations) yield better performance and stability?
• RQ4 What are the practical hyperparameters and computational costs when applying VAT to neural networks?
• RQ5 How does VAT compare to state-of-the-art semi-supervised methods on standard benchmarks?

Key findings

• VAT achieves superior or competitive performance on MNIST and CIFAR-10 compared to contemporary regularization methods.
• VAT extended with entropy minimization achieves state-of-the-art results for semi-supervised learning on SVHN and CIFAR-10.
• The VAT algorithm requires only two scalar hyperparameters and has low computational overhead (approximately three times the cost of standard training).
• A power-iteration-based approximation enables efficient computation of virtual adversarial perturbations with minimal additional backpropagation.
• VAT outperforms random perturbation training (RPT) by focusing smoothing in the most anisotropic direction of the model’s local output distribution.
• The method is applicable to any differentiable model and supports semi-supervised learning without label information on unlabeled data.