[Paper Review] Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks

Lechao Xiao, Yasaman Bahri+3 more|arXiv (Cornell University)|Jun 14, 2018

Neural Networks and ApplicationsComputer Science175 citations

TL;DR

The paper develops a mean field theory for CNNs to identify initialization schemes that enable training vanilla CNNs with up to 10,000 layers without architectural tricks. It introduces orthogonal and Delta-Orthogonal initializations to achieve dynamical isometry and balanced signal propagation across frequencies.

ABSTRACT

In recent years, state-of-the-art methods in computer vision have utilized increasingly deep convolutional neural network architectures (CNNs), with some of the most successful models employing hundreds or even thousands of layers. A variety of pathologies such as vanishing/exploding gradients make training such deep networks challenging. While residual connections and batch normalization do enable training at these depths, it has remained unclear whether such specialized architecture designs are truly necessary to train deep CNNs. In this work, we demonstrate that it is possible to train vanilla CNNs with ten thousand layers or more simply by using an appropriate initialization scheme. We derive this initialization scheme theoretically by developing a mean field theory for signal propagation and by characterizing the conditions for dynamical isometry, the equilibration of singular values of the input-output Jacobian matrix. These conditions require that the convolution operator be an orthogonal transformation in the sense that it is norm-preserving. We present an algorithm for generating such random initial orthogonal convolution kernels and demonstrate empirically that they enable efficient training of extremely deep architectures.

Motivation & Objective

• Motivate whether extremely deep vanilla CNNs can be trained without residual connections or batch normalization by proper initialization.
• Derive a mean field theory for signal propagation in CNNs and identify conditions for dynamical isometry.
• Develop practical initialization schemes (orthogonal and Delta-Orthogonal) that realize these conditions.
• Demonstrate experimentally that ultra-deep CNNs can be trained on standard benchmarks with the proposed initializations.

Proposed method

• Derive a forward propagation covariance recursion for CNNs and study its fixed points and stability.
• Analyze the layer-to-layer Jacobian and its singular value distribution to connect to dynamical isometry.
• Construct random orthogonal convolutional kernels using a wavelet-inspired method.
• Introduce Delta-Orthogonal initialization by concentrating variance at the kernel center to balance propagation across Fourier modes.
• Provide empirical validation on MNIST and CIFAR-10 showing training speedups and depth limits.

Experimental results

Research questions

• RQ1 Can vanilla CNNs be trained at depths of 10,000 layers without residual connections or batch normalization with an appropriate initialization?
• RQ2 What initialization conditions ensure stable forward signal propagation and dynamical isometry in CNNs?
• RQ3 How do spatial frequency modes propagate in very deep CNNs, and how can initialization balance them?
• RQ4 Do orthogonal and Delta-Orthogonal initializations improve training speed and generalization for ultra-deep CNNs?

Key findings

• Vanilla CNNs can be trained up to 10,000 layers using a theoretically motivated initialization scheme.
• Orthogonal kernels dramatically improve training speed and conditioning of the end-to-end Jacobian.
• CNNs exhibit multiple depth scales for different spatial frequencies, and Delta-Orthogonal initialization allows balanced propagation across all frequencies.
• Non-uniform weight variance across the kernel alters mode propagation and can degrade generalization for very deep networks.
• Delta-Orthogonal initialization achieves arbitrary depth with preserved performance in experiments, whereas generic Gaussian initializations fail.
• Empirical results on MNIST and CIFAR-10 show training progress and depth-related generalization effects consistent with the theory.