[Paper Review] Swapout: Learning an ensemble of deep architectures

Saurabh Singh, Derek Hoiem+1 more|arXiv (Cornell University)|May 20, 2016

Advanced Neural Network Applications18 references105 citations

TL;DR

Swapout is a stochastic training method that generalizes dropout and stochastic depth to sample an ensemble of architectures at unit- and layer-level, improving accuracy over ResNets of the same depth and enabling very wide, shallower networks to match deep models.

ABSTRACT

We describe Swapout, a new stochastic training method, that outperforms ResNets of identical network structure yielding impressive results on CIFAR-10 and CIFAR-100. Swapout samples from a rich set of architectures including dropout, stochastic depth and residual architectures as special cases. When viewed as a regularization method swapout not only inhibits co-adaptation of units in a layer, similar to dropout, but also across network layers. We conjecture that swapout achieves strong regularization by implicitly tying the parameters across layers. When viewed as an ensemble training method, it samples a much richer set of architectures than existing methods such as dropout or stochastic depth. We propose a parameterization that reveals connections to exiting architectures and suggests a much richer set of architectures to be explored. We show that our formulation suggests an efficient training method and validate our conclusions on CIFAR-10 and CIFAR-100 matching state of the art accuracy. Remarkably, our 32 layer wider model performs similar to a 1001 layer ResNet model.

Motivation & Objective

• Motivate regularization and architectural diversity in deep networks beyond dropout and stochastic depth.
• Develop a generalized stochastic training framework (Swapout) that samples from a rich set of architectures.
• Evaluate Swapout against ResNets and baseline stochastic methods on CIFAR-10 and CIFAR-100.
• Demonstrate that wider, shallower Swapout models can match or exceed very deep residual networks.

Proposed method

• Define Swapout as a per-unit stochastic choice among multiple options including 0, X, F(X), and X+F(X).
• Show that Swapout generalizes dropout and stochastic depth as special cases.
• Provide a Lipschitz-stability discussion linking Swapout to SGD stability similar to dropout.
• Compare inference approaches: deterministic (expectations) vs. stochastic (sampling multiple network instances).
• Experiment with ResNet-like blocks on CIFAR-10/100, using deterministic and stochastic inference, and varying network width and depth.
• Present parameter-efficient results demonstrating wider, shallower Swapout networks can rival very deep ResNets.

Experimental results

Research questions

• RQ1 Can Swapout improve accuracy over ResNets of equivalent depth on CIFAR-10 and CIFAR-100?
• RQ2 Does increasing network width with Swapout provide gains comparable to deeper architectures?
• RQ3 How do different stochastic training schedules (per-layer Bernoulli parameters) affect performance?
• RQ4 Is stochastic inference (sampling multiple forward passes) advantageous for Swapout over deterministic inference?
• RQ5 What is the relationship between parameter efficiency and performance for Swapout compared to baseline methods?

Key findings

• Swapout improves accuracy over comparable ResNet baselines on CIFAR-10 and CIFAR-100.
• A 32-layer wider Swapout model matches the performance of a 1001-layer ResNet on both CIFAR datasets.
• Increasing width with Swapout yields substantial gains and can outperform deeper ResNets with more parameters.
• Stochastic inference (averaging predictions from multiple samples) consistently outperforms deterministic inference in Swapout.
• Different stochastic training schedules significantly affect performance, with less randomness in early layers generally performing better.
• Swapout achieves parameter-efficient gains, sometimes outperforming deeper models with fewer parameters.