[Paper Review] Deep Networks with Stochastic Depth

Gao Huang, Yu Sun+3 more|arXiv (Cornell University)|Mar 30, 2016

Advanced Neural Network Applications28 references287 citations

TL;DR

This paper proposes stochastic depth to train very deep ResNet-like networks by randomly dropping layers during training, enabling deep models (over 1000 layers) with faster training and improved test accuracy, and treating training as an implicit ensemble of varying depths.

ABSTRACT

Very deep convolutional networks with hundreds of layers have led to significant reductions in error on competitive benchmarks. Although the unmatched expressiveness of the many layers can be highly desirable at test time, training very deep networks comes with its own set of challenges. The gradients can vanish, the forward flow often diminishes, and the training time can be painfully slow. To address these problems, we propose stochastic depth, a training procedure that enables the seemingly contradictory setup to train short networks and use deep networks at test time. We start with very deep networks but during training, for each mini-batch, randomly drop a subset of layers and bypass them with the identity function. This simple approach complements the recent success of residual networks. It reduces training time substantially and improves the test error significantly on almost all data sets that we used for evaluation. With stochastic depth we can increase the depth of residual networks even beyond 1200 layers and still yield meaningful improvements in test error (4.91% on CIFAR-10).

Motivation & Objective

• Motivate and address the training challenges of very deep CNNs (vanishing gradients, diminished forward flow, long training times).
• Propose a simple, compatible training procedure (stochastic depth) built on residual networks to shorten depth during training while preserving full depth at test time.
• Show that stochastic depth reduces training time and improves test error across multiple datasets (CIFAR-10/100, SVHN, ImageNet).
• Demonstrate that the method acts as an implicit ensemble and provides regularization effects similar to dropout, benefiting networks with batch normalization.

Proposed method

• Introduce a Bernoulli variable b_l for each ResBlock to indicate if it is active during a training pass (b_l in {0,1}).
• Define survival probability p_l for each block and bypass inactive blocks by replacing f_l(H_{l-1}) with 0 in the block’s path, effectively using the identity skip connection when inactive.
• Use a linear decay rule p_l = 1 - (l/L)(1 - p_L) starting from p_0 = 1 to p_L (typically p_L = 0.5).
• Train very deep ResNets by reducing effective depth on each mini-batch and sampling different sub-networks; during testing keep full depth but scale outputs by p_l to reflect training participation (equation 5).
• Argue that stochastic depth yields faster training (depth-averaged forward/backward passes) and improves generalization by implicit ensemble and regularization effects.
• Empirically evaluate on CIFAR-10/100, SVHN, and ImageNet, including experiments with 1202-layer networks where stochastic depth yields gains over standard ResNets.

Experimental results

Research questions

• RQ1 Can stochastic depth enable training of networks deeper than 1000 layers while maintaining or improving test accuracy?
• RQ2 Does shortening the network during training reduce training time without sacrificing performance?
• RQ3 How does stochastic depth affect gradient flow and training dynamics compared to constant-depth ResNets?
• RQ4 Does the method act as an effective regularizer and implicitly ensemble multiple depths?

Key findings

• Training with stochastic depth yields substantial reductions in training time (about 25% on CIFAR-10/100 and SVHN).
• ResNets with stochastic depth achieve lower test error than constant-depth counterparts across CIFAR-10 (5.25% vs 6.41%), CIFAR-100 (24.98% vs 27.76%), and SVHN (1.75% vs 1.80%).
• On CIFAR-10, a 1202-layer ResNet with stochastic depth achieves 4.91% test error, a new record at the time, and shows improvements over the 110-layer version.
• On CIFAR-100, stochastic depth reduces error to 24.98% from 27.76% (state-of-the-art single-model performance at the time).
• ImageNet results show the method can be applied to very deep networks (152-layer ResNet); with stochastic depth, a final validation error of 21.78% and 21.98% were reported, indicating potential gains with longer training on larger models.
• The method strengthens gradient flow (larger gradient magnitudes after learning-rate drops) and behaves as a strong regularizer akin to dropout, even with Batch Normalization.