[Paper Review] DiracNets: Training Very Deep Neural Networks Without Skip-Connections

Sergey Zagoruyko, Nikos Komodakis|arXiv (Cornell University)|Jun 1, 2017

Advanced Neural Network ApplicationsComputer Science16 references75 citations

TL;DR

DiracNets use a Dirac weight parameterization to train very deep plain networks without explicit skip-connections, achieving near-ResNet/WRN performance and folding into simple convolution-ReLU chains at inference.

ABSTRACT

Deep neural networks with skip-connections, such as ResNet, show excellent performance in various image classification benchmarks. It is though observed that the initial motivation behind them - training deeper networks - does not actually hold true, and the benefits come from increased capacity, rather than from depth. Motivated by this, and inspired from ResNet, we propose a simple Dirac weight parameterization, which allows us to train very deep plain networks without explicit skip-connections, and achieve nearly the same performance. This parameterization has a minor computational cost at training time and no cost at all at inference, as both Dirac parameterization and batch normalization can be folded into convolutional filters, so that network becomes a simple chain of convolution-ReLU pairs. We are able to match ResNet-1001 accuracy on CIFAR-10 with 28-layer wider plain DiracNet, and closely match ResNets on ImageNet. Our parameterization also mostly eliminates the need of careful initialization in residual and non-residual networks. The code and models for our experiments are available at https://github.com/szagoruyko/diracnets

Motivation & Objective

• Understand limitations of skip-connections and deepening in image classification.
• Propose Dirac weight parameterization to allow very deep plain networks to train end-to-end.
• Demonstrate DiracNet performance on CIFAR and ImageNet against ResNet and WRN.
• Show how Dirac parameterization interacts with initialization and can be folded for inference.

Proposed method

• Introduce Dirac parameterization where weight W_hat = diag(a)I + W (with optional weight normalization W_hat = diag(a)I + diag(b)W_norm).
• Initialize a ~ 1 and b ~ 0.1; W initialized from N(0,1); no L2 regularization on a,b.
• Use weight normalization and folding to train very deep plain networks; compare to ResNet/WRN on CIFAR and ImageNet.
• Relate Dirac parameterization to ResNet by showing implicit skip connections and discuss order of nonlinearities.
• Evaluate on CIFAR withplain and DiracNet variants, and on ImageNet with DiracNet-18/34 versus ResNet-18/34.
• Demonstrate that DiracNet can train end-to-end without layer-wise pretraining and can be folded into a VGG-like chain at inference.

Experimental results

Research questions

• RQ1 Can Dirac parameterization enable training of hundreds of layers without explicit skip connections?
• RQ2 How does DiracNet performance compare to ResNet and Wide ResNet on CIFAR-10/100 and ImageNet?
• RQ3 Does Dirac parameterization reduce sensitivity to initialization and allow folding into a simple convolution-ReLU chain at test time?
• RQ4 What is the impact of network width and depth on DiracNets versus traditional residual networks?

Key findings

• DiracNets enable training of very deep plain networks (hundreds of layers) with competitive performance.
• DiracNet-28-10 achieves 4.75% top-1 / 21.54% top-5 on CIFAR-10/100 with 36.5M parameters, closely approaching WRN-28-10.
• On CIFAR, plain DiracNets outperform other plain networks and approach ResNet/WRN performance; deeper DiracNets gain accuracy where plain nets fail.
• On ImageNet, DiracNet-18/34 nearly match ResNet-18/34 with similar parameter counts.
• Dirac parameterization eliminates the need for careful initialization in ResNet-like nets when used with this framework.
• Dirac parameterized filters can be folded into a single weight vector, yielding a simple VGG-like chain of conv-ReLU blocks at inference.