[Paper Review] Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks

Wei Hu, Lechao Xiao+1 more|arXiv (Cornell University)|Jan 16, 2020

Speech Recognition and Synthesis30 references60 citations

TL;DR

The paper proves that orthogonal initialization speeds up convergence of gradient descent in deep linear networks, with width requirements independent of depth, unlike Gaussian initialization whose width must grow with depth.

ABSTRACT

The selection of initial parameter values for gradient-based optimization of deep neural networks is one of the most impactful hyperparameter choices in deep learning systems, affecting both convergence times and model performance. Yet despite significant empirical and theoretical analysis, relatively little has been proved about the concrete effects of different initialization schemes. In this work, we analyze the effect of initialization in deep linear networks, and provide for the first time a rigorous proof that drawing the initial weights from the orthogonal group speeds up convergence relative to the standard Gaussian initialization with iid weights. We show that for deep networks, the width needed for efficient convergence to a global minimum with orthogonal initializations is independent of the depth, whereas the width needed for efficient convergence with Gaussian initializations scales linearly in the depth. Our results demonstrate how the benefits of a good initialization can persist throughout learning, suggesting an explanation for the recent empirical successes found by initializing very deep non-linear networks according to the principle of dynamical isometry.

Motivation & Objective

• Evaluate how initialization schemes affect gradient descent convergence in deep linear networks.
• Provide a rigorous comparison between orthogonal and Gaussian initializations on convergence rates.
• Characterize width-depth trade-offs under different initializations to achieve efficient training.

Proposed method

• Analyze gradient descent dynamics for an L-layer deep linear network with weight matrices W1,...,WL.
• Initialize weights orthogonally with specific scaling so that E||f(x;W...W1)||^2 = ||x||^2.
• Prove that with orthogonal initialization and sufficient width m, the training loss converges to the global minimum at a rate independent of depth.
• Contrast with Gaussian initialization, showing that for Gaussian initialization the necessary width scales linearly with depth for efficient convergence.
• Use a time-varying PSD matrix framework to track network output evolution and bound eigenvalues during training.

Experimental results

Research questions

• RQ1 Does orthogonal initialization accelerate convergence of gradient descent on deep linear networks compared to Gaussian initialization?
• RQ2 How should layer widths (the hidden width m) scale with depth L to guarantee efficient convergence under orthogonal vs Gaussian initialization?
• RQ3 What is the impact of initialization on the dependency of convergence rate on depth and width?
• RQ4 Can orthogonal initialization yield depth-independent width requirements for efficient optimization?

Key findings

• Orthogonal initialization enables efficient convergence with width m that does not depend on depth L.
• Gaussian initialization requires width that scales linearly with depth for efficient convergence (unless depth is small).
• With orthogonal initialization, the loss-to-optimal ratio decreases geometrically with iterations under suitable learning rate.
• Gaussian initialization can lead to exponentially long convergence times when width is insufficient relative to depth.
• Empirical experiments corroborate the theoretical results, showing depth-independent width requirements for orthogonal initialization and depth-dependent width for Gaussian initialization.