[Paper Review] How to Start Training: The Effect of Initialization and Architecture

Boris Hanin, David Rolnick|arXiv (Cornell University)|Mar 5, 2018

Adversarial Robustness in Machine Learning32 references50 citations

TL;DR

The paper rigorously analyzes two early-training failure modes in ReLU networks and shows how proper initialization and architecture—especially for ResNets—prevent these failures, enabling training of deeper networks. It provides theoretical results and empirical validation across fully connected, convolutional, and residual architectures.

ABSTRACT

We identify and study two common failure modes for early training in deep ReLU nets. For each we give a rigorous proof of when it occurs and how to avoid it, for fully connected and residual architectures. The first failure mode, exploding/vanishing mean activation length, can be avoided by initializing weights from a symmetric distribution with variance 2/fan-in and, for ResNets, by correctly weighting the residual modules. We prove that the second failure mode, exponentially large variance of activation length, never occurs in residual nets once the first failure mode is avoided. In contrast, for fully connected nets, we prove that this failure mode can happen and is avoided by keeping constant the sum of the reciprocals of layer widths. We demonstrate empirically the effectiveness of our theoretical results in predicting when networks are able to start training. In particular, we note that many popular initializations fail our criteria, whereas correct initialization and architecture allows much deeper networks to be trained.

Motivation & Objective

• Identify failure modes that block early training in deep ReLU networks (FM1 and FM2).
• Provide rigorous conditions on initialization and architecture to avoid FM1 and FM2 in FC, Conv, and ResNet architectures.
• Demonstrate empirically how correct initialization and architecture predict training feasibility and depth.
• Compare behavior across fully connected, convolutional, and residual networks to explain empirical training success of ResNets.

Proposed method

• Define and analyze two failure modes: FM1 (mean activation length grows/shrinks exponentially with depth) and FM2 (variance of activation lengths across layers grows exponentially).
• Prove that FM1 can be avoided by initializing weights with symmetric distributions of variance 2/fan-in (and scaling residual modules in ResNets).
• Show that FM2 is avoided in ResNets once FM1 is avoided, while in fully connected nets FM2 depends on architecture via the sum of reciprocals of layer widths.
• Derive and state formal theorems (Theorem 1–Theorem 6) describing conditions under which FM1 and FM2 occur or are prevented in FC, Conv, and ResNet architectures.
• Extend results to convolutional architectures by replacing fan-in with the appropriate fan-in for conv layers and demonstrating similar behavior empirically.

Experimental results

Research questions

• RQ1 Under what initialization and architectural conditions do FM1 and FM2 occur in deep ReLU networks?
• RQ2 How do FC, Conv, and ResNet architectures differ in their propensity for FM2, and how does this relate to training feasibility of deep networks?
• RQ3 Can proper scaling of residual modules and weight variance enable training of significantly deeper ResNets?
• RQ4 Do empirical activation lengths at initialization reliably predict early training performance across architectures?

Key findings

• Initializing weights from a symmetric distribution with variance 2/fan-in prevents exploding/vanishing mean activation length (FM1) in FC and Conv nets.
• Correctly scaling residual modules in ResNets prevents FM1, and FM2 cannot occur in ResNets once FM1 is avoided (Corollary/Theorem 6).
• For fully connected and convolutional nets, FM2 depends on architecture and is mitigated by wider layers or by linear growth of width with depth; constant-width networks require width to grow roughly linearly with depth to avoid FM2.
• For residual networks, FM2 is largely architecture-independent once FM1 is avoided; residual modules weighted appropriately ensure stable activation lengths across depth.
• Empirically, networks initialized with correct variance and architecture exhibit successful training for greater depths, while popular initializations often fail FM1.