[Paper Review] Data-dependent Initializations of Convolutional Neural Networks

Philipp Krähenbühl, Carl Doersch+2 more|arXiv (Cornell University)|Nov 21, 2015

Neural Networks and Applications90 citations

TL;DR

This paper proposes a data-dependent initialization method for Convolutional Neural Networks (CNNs) that scales network weights based on statistical properties of training data to ensure uniform gradient flow across layers. By analyzing activation statistics and applying layer-wise rescaling, the method accelerates training convergence and matches or exceeds state-of-the-art self-supervised pre-training performance on image classification and object detection, reducing pre-training time by up to 1,000x while improving generalization on small datasets.

ABSTRACT

Convolutional Neural Networks spread through computer vision like a wildfire, impacting almost all visual tasks imaginable. Despite this, few researchers dare to train their models from scratch. Most work builds on one of a handful of ImageNet pre-trained models, and fine-tunes or adapts these for specific tasks. This is in large part due to the difficulty of properly initializing these networks from scratch. A small miscalibration of the initial weights leads to vanishing or exploding gradients, as well as poor convergence properties. In this work we present a fast and simple data-dependent initialization procedure, that sets the weights of a network such that all units in the network train at roughly the same rate, avoiding vanishing or exploding gradients. Our initialization matches the current state-of-the-art unsupervised or self-supervised pre-training methods on standard computer vision tasks, such as image classification and object detection, while being roughly three orders of magnitude faster. When combined with pre-training methods, our initialization significantly outperforms prior work, narrowing the gap between supervised and unsupervised pre-training.

Motivation & Objective

• To address the challenge of poor convergence and vanishing/exploding gradients when training deep CNNs from scratch.
• To identify data-driven statistical properties of activations that significantly affect generalization performance on small datasets.
• To develop a simple, fast initialization method that enables faster and more stable training without requiring pre-training or architectural modifications.
• To reduce reliance on ImageNet pre-training by improving initialization quality for downstream tasks with limited labeled data.

Proposed method

• The method computes the mean and standard deviation of feature maps across a small batch of training data to estimate activation statistics per layer.
• It rescales the weights of each convolutional layer such that the output activation variance is approximately equal across all layers.
• The initialization enforces that the gradient norm is roughly uniform across layers by adjusting the scale of weight matrices based on input statistics.
• It uses a k-means clustering of feature maps to estimate the effective receptive field and refine the scaling factor for each layer.
• The approach is applied before training and requires only a single forward pass over a small data subset, making it computationally efficient.
• It does not require backpropagation or additional loss terms, making it compatible with any CNN architecture.

Experimental results

Research questions

• RQ1 Can data-driven initialization improve training convergence and generalization in CNNs without pre-training?
• RQ2 How do statistical properties of feature activations influence gradient flow and model performance during fine-tuning?
• RQ3 Can a simple, fast initialization method match or surpass the performance of complex self-supervised pre-training methods on downstream tasks?
• RQ4 Does data-dependent initialization reduce the need for ImageNet pre-training in low-data regimes?
• RQ5 Can this method enable stable training of deeper networks without architectural modifications like auxiliary classifiers?

Key findings

• The proposed initialization reduces pre-training time by three orders of magnitude—down to 54 seconds—while achieving 56.6% top-1 accuracy on PASCAL VOC2007 image classification, matching state-of-the-art self-supervised methods.
• When combined with unsupervised pre-training (e.g., Doersch et al., 2015), the method achieves 65.3% mAP on PASCAL VOC2007 detection, outperforming prior work.
• On ImageNet, the method accelerates training convergence by up to 10x in early training stages, with faster error reduction in the first 10,000 iterations compared to standard initialization.
• The k-means-based initialization outperforms random initialization and achieves comparable performance to the reference ImageNet-pretrained model after 100k iterations.
• Removing LRN layers in CaffeNet did not degrade performance when using the proposed initialization, suggesting normalization layers may be redundant with proper weight scaling.
• In GoogLeNet, the method enabled training with a single classifier (no auxiliary heads), achieving similar convergence speed to the original three-head architecture, demonstrating robustness in deep networks.