[Paper Review] All you need is a good init

Dmytro Mishkin, Jiřı́ Matas|arXiv (Cornell University)|Nov 19, 2015

Advanced Neural Network Applications16 references206 citations

TL;DR

This paper proposes Layer-Sequential Unit-Variance (LSUV) initialization, a simple two-step method that first uses orthonormal weight matrices and then normalizes each layer's output variance to one. It enables training of very deep networks with standard SGD, achieving state-of-the-art or near-state-of-the-art accuracy on MNIST, CIFAR, and ImageNet with faster convergence than complex methods like FitNets and Highway Networks.

ABSTRACT

Layer-sequential unit-variance (LSUV) initialization - a simple method for weight initialization for deep net learning - is proposed. The method consists of the two steps. First, pre-initialize weights of each convolution or inner-product layer with orthonormal matrices. Second, proceed from the first to the final layer, normalizing the variance of the output of each layer to be equal to one. Experiment with different activation functions (maxout, ReLU-family, tanh) show that the proposed initialization leads to learning of very deep nets that (i) produces networks with test accuracy better or equal to standard methods and (ii) is at least as fast as the complex schemes proposed specifically for very deep nets such as FitNets (Romero et al. (2015)) and Highway (Srivastava et al. (2015)). Performance is evaluated on GoogLeNet, CaffeNet, FitNets and Residual nets and the state-of-the-art, or very close to it, is achieved on the MNIST, CIFAR-10/100 and ImageNet datasets.

Motivation & Objective

• To address the challenge of training very deep neural networks from scratch using standard stochastic gradient descent (SGD), which often fails due to poor weight initialization.
• To develop a simple, general, and efficient initialization method that avoids the need for complex training schemes or auxiliary networks.
• To evaluate whether a straightforward variance normalization procedure can match or surpass the performance of sophisticated deep learning architectures like FitNets and Highway Networks.
• To investigate the impact of initialization on training speed and final accuracy across different activation functions and network architectures.
• To compare LSUV initialization with batch normalization in terms of convergence speed and final performance, particularly on large-scale datasets like ImageNet.

Proposed method

• Pre-initialize all convolutional and fully connected layer weights using orthonormal matrices to ensure stable initial signal propagation.
• Process the network layer-by-layer from input to output, normalizing the activation variance of each layer to exactly one using a forward pass with a small batch of data.
• Apply the variance normalization only once before training begins, making it a one-time preprocessing step.
• Use singular value decomposition (SVD) to generate orthonormal weight matrices, ensuring that the initial weight distribution maintains unit variance across layers.
• Combine the orthonormal initialization with layer-wise variance normalization to stabilize gradient flow and prevent vanishing/exploding gradients.
• Train the network using standard SGD without additional normalization layers or complex optimization schemes.

Experimental results

Research questions

• RQ1 Can a simple, one-time weight initialization procedure enable end-to-end training of very deep networks with standard SGD?
• RQ2 Does LSUV initialization outperform or match the performance of complex deep learning architectures like FitNets and Highway Networks on standard benchmarks?
• RQ3 How does LSUV compare to batch normalization in terms of convergence speed and final accuracy, especially on large-scale datasets like ImageNet?
• RQ4 Does LSUV work effectively across different activation functions such as ReLU, maxout, and tanh?
• RQ5 Can LSUV reduce the training time and improve convergence without introducing computational overhead during training?

Key findings

• LSUV initialization achieves state-of-the-art or near-state-of-the-art test accuracy on MNIST, CIFAR-10/100, and ImageNet, outperforming or matching complex methods like FitNets and Highway Networks.
• On GoogLeNet, LSUV-initialized networks converge faster and achieve a final top-1 accuracy of 68.0% compared to 67.2% for the original initialization, demonstrating consistent improvement.
• On CaffeNet, LSUV reduces the initial flat-loss phase from 0.5 to 0.05 epochs but is eventually overtaken by the original network, with a 1.3% lower final accuracy, though the reason remains unexplained.
• LSUV initialization is as fast as batch normalization in terms of wall-clock time, with LSUV networks matching BN performance on CIFAR-10 while avoiding the 30% computational overhead per iteration.
• The method is robust across different activation functions, including ReLU, maxout, and tanh, showing consistent performance gains.
• The computational cost of LSUV initialization is minimal—up to 3.5 minutes for CaffeNet—making it negligible compared to training time.