[Paper Review] Semi-Supervised Learning with Ladder Networks

Antti Rasmus, Harri Valpola+3 more|arXiv (Cornell University)|Jul 9, 2015

Domain Adaptation and Few-Shot Learning39 references596 citations

TL;DR

This paper proposes a semi-supervised deep learning framework that integrates supervised learning with unsupervised denoising autoencoding via Ladder networks, using skip connections and layer-wise reconstruction to improve generalization. It achieves state-of-the-art performance on MNIST, CIFAR-10, and permutation-invariant MNIST with minimal labeled data, outperforming prior methods in both semi-supervised and fully supervised settings.

ABSTRACT

We combine supervised learning with unsupervised learning in deep neural networks. The proposed model is trained to simultaneously minimize the sum of supervised and unsupervised cost functions by backpropagation, avoiding the need for layer-wise pre-training. Our work builds on the Ladder network proposed by Valpola (2015), which we extend by combining the model with supervision. We show that the resulting model reaches state-of-the-art performance in semi-supervised MNIST and CIFAR-10 classification, in addition to permutation-invariant MNIST classification with all labels.

Motivation & Objective

• To develop a deep neural network architecture that simultaneously performs supervised and unsupervised learning without requiring layer-wise pre-training.
• To improve generalization in semi-supervised learning by leveraging unlabeled data through a denoising autoencoding objective at every hidden layer.
• To demonstrate that the unsupervised component enhances performance not only in low-label regimes but also in fully supervised settings.
• To enable scalability and computational efficiency in deep architectures by using local, layer-wise unsupervised objectives.

Proposed method

• The model uses a deep feedforward network with skip connections from each encoder layer to its corresponding decoder layer, forming a 'ladder' structure.
• Each hidden layer is trained to reconstruct its own noisy version, using a denoising autoencoder objective that encourages robust, invariant representations.
• The decoder is structured to mirror the encoder, with vertical mappings reversed, enabling gradient flow through both encoder and decoder paths.
• The network is trained end-to-end via backpropagation using a combined loss function that sums supervised cross-entropy and unsupervised reconstruction losses.
• Batch normalization is applied to accelerate convergence and stabilize training.
• The method is compatible with standard architectures like MLPs and CNNs, requiring minimal architectural modifications.

Experimental results

Research questions

• RQ1 Can a unified training procedure combining supervised and unsupervised learning improve performance on semi-supervised image classification tasks?
• RQ2 Does the inclusion of layer-wise denoising autoencoding improve generalization in deep networks, especially when labeled data is scarce?
• RQ3 Can the unsupervised component enhance performance even in fully supervised settings, such as permutation-invariant MNIST?
• RQ4 How does the Ladder network's architecture with skip connections and local unsupervised objectives compare to traditional pre-training or end-to-end training?

Key findings

• The model achieves state-of-the-art performance on semi-supervised MNIST with only 100 labeled examples, significantly outperforming prior methods.
• On CIFAR-10, the model achieves a test error of 1.4% with 4,000 labeled images, setting a new SOTA in semi-supervised learning.
• For the permutation-invariant MNIST task, the model achieves a new state-of-the-art result even with full-label supervision, indicating no negative interference from the unsupervised objective.
• The method improves performance across multiple deep architectures, including both MLPs and CNNs, without requiring architectural redesign.
• The training process is computationally efficient, with only a small multiplicative factor increase in computation per update compared to standard supervised learning.
• The model converges quickly and reliably, especially when combined with batch normalization, demonstrating strong practical utility.