[Paper Review] Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning

Mehdi S. M. Sajjadi, Mehran Javanmardi+1 more|arXiv (Cornell University)|Jun 14, 2016

Domain Adaptation and Few-Shot Learning37 references551 citations

TL;DR

The paper introduces an unsupervised loss that regularizes CNNs by minimizing prediction differences across multiple stochastic passes of the same sample, enabling improved semi-supervised learning with limited labeled data.

ABSTRACT

Effective convolutional neural networks are trained on large sets of labeled data. However, creating large labeled datasets is a very costly and time-consuming task. Semi-supervised learning uses unlabeled data to train a model with higher accuracy when there is a limited set of labeled data available. In this paper, we consider the problem of semi-supervised learning with convolutional neural networks. Techniques such as randomized data augmentation, dropout and random max-pooling provide better generalization and stability for classifiers that are trained using gradient descent. Multiple passes of an individual sample through the network might lead to different predictions due to the non-deterministic behavior of these techniques. We propose an unsupervised loss function that takes advantage of the stochastic nature of these methods and minimizes the difference between the predictions of multiple passes of a training sample through the network. We evaluate the proposed method on several benchmark datasets.

Motivation & Objective

• Motivate leveraging unlabeled data to improve CNN generalization when labeled data are scarce.
• Propose an unsupervised loss that exploits non-deterministic transformations and perturbations during training.
• Combine transformation/stability loss with a mutual-exclusivity loss to guide learning without labels.
• Demonstrate effectiveness across multiple datasets and architectures using two CNN implementations.

Proposed method

• Define an unsupervised loss that minimizes the mean squared difference between predictions from multiple passes of the same sample under random transformations T^j(x_i) and network perturbations.
• Represent the transformation/stability loss l_U^TS as the sum of squared differences across all pairs of passes (Equation 1).
• Introduce a mutual-exclusivity loss l_U^ME to prevent trivial prediction vectors (Equation 2).
• Combine losses into l_U = λ1 l_U^ME + λ2 l_U^TS (Equation 3).
• Show compatibility with supervised losses and batch training by incorporating replicated samples within mini-batches.
• Discuss two CNN implementations (cuda-convnet AlexNet variant and sparse convolutional networks with fractional max-pooling) and how n (passes per sample) is chosen (n=4 or n=5) based on framework constraints.

Experimental results

Research questions

• RQ1 Can an unsupervised loss that enforces consistency across stochastic transformations and network perturbations improve semi-supervised CNN performance?
• RQ2 How does coupling a mutual-exclusivity loss with transformation/stability loss affect learning when labels are scarce?
• RQ3 Do the proposed losses generalize across datasets (MNIST, SVHN, NORB, CIFAR-10/100, ImageNet) and architectures?
• RQ4 What is the impact of the number of passes and data augmentation strategies on convergence and accuracy?

Key findings

• The transformation/stability and mutual-exclusivity losses consistently improve test accuracy when labeled data are limited across multiple datasets.
• The combination of both unsupervised losses yields greater gains than either loss alone, especially with very small labeled sets.
• Using replicated passes reduces error rates significantly (e.g., notable gains when using 1%–5% labeled data on SVHN and NORB).
• The approach achieves competitive or state-of-the-art results on several benchmarks (e.g., CIFAR-10/100, ImageNet) when combined with data augmentation and dropout/RP.