[Paper Review] Distributional Smoothing with Virtual Adversarial Training

Takeru Miyato, Shin‐ichi Maeda+3 more|arXiv (Cornell University)|Jan 1, 2016

Adversarial Robustness in Machine Learning266 citations

TL;DR

This paper introduces Virtual Adversarial Training (VAT), a semi-supervised learning method that enforces local distributional smoothness by optimizing model robustness against unlabeled data perturbations using only model predictions, not labels. VAT achieves state-of-the-art performance on MNIST, SVHN, and NORB, outperforming existing methods except for highly complex generative models.

ABSTRACT

Abstract: We propose local distributional smoothness (LDS), a new notion of smoothness for statistical model that can be used as a regularization term to promote the smoothness of the model distribution. We named the LDS based regularization as virtual adversarial training (VAT). The LDS of a model at an input datapoint is defined as the KL-divergence based robustness of the model distribution against local perturbation around the datapoint. VAT resembles adversarial training, but distinguishes itself in that it determines the adversarial direction from the model distribution alone without using the label information, making it applicable to semi-supervised learning. The computational cost for VAT is relatively low. For neural network, the approximated gradient of the LDS can be computed with no more than three pairs of forward and back propagations. When we applied our technique to supervised and semi-supervised learning for the MNIST dataset, it outperformed all the training methods other than the current state of the art method, which is based on a highly advanced generative model. We also applied our method to SVHN and NORB, and confirmed our method's superior performance over the current state of the art semi-supervised method applied to these datasets.

Motivation & Objective

• To develop a regularization technique that improves model generalization by enforcing local distributional smoothness without relying on label information.
• To enable effective semi-supervised learning by leveraging unlabeled data through robustness to local perturbations.
• To design a computationally efficient method that requires minimal additional computation beyond standard backpropagation.
• To outperform existing semi-supervised learning methods on benchmark vision datasets like MNIST, SVHN, and NORB.

Proposed method

• Proposes local distributional smoothness (LDS), defined as the KL divergence between the model's output distribution and its distribution after a small perturbation around a given input.
• Uses the model's own predictive distribution to determine the most damaging perturbation direction, avoiding reliance on labels.
• Computes the approximate gradient of LDS using at most three forward and backward passes, enabling efficient optimization.
• Applies the regularization during training by minimizing the LDS loss on unlabeled data, promoting robustness and generalization.
• Integrates the LDS regularization into both supervised and semi-supervised learning frameworks.

Experimental results

Research questions

• RQ1 Can enforcing local distributional smoothness improve model generalization in semi-supervised learning?
• RQ2 Can adversarial perturbations be generated without label information to improve robustness?
• RQ3 Does the proposed method achieve better performance than existing semi-supervised learning baselines on standard vision benchmarks?
• RQ4 How does the computational cost of the method compare to standard training and other adversarial training approaches?

Key findings

• VAT achieved performance on MNIST that surpassed all training methods except the current state-of-the-art method based on a highly advanced generative model.
• On SVHN and NORB, VAT outperformed the then-current state-of-the-art semi-supervised learning method.
• The method required no more than three forward and backward passes per update, making it computationally efficient.
• The model's robustness to local perturbations significantly improved generalization, especially in low-labeled-data regimes.