[Paper Review] A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

Sanjeev Arora, Nadav Cohen+2 more|arXiv (Cornell University)|Oct 4, 2018

Stochastic Gradient Optimization Techniques39 references113 citations

TL;DR

The paper proves gradient descent converges linearly to the global minimum for deep linear networks trained with 2-norm loss on whitened data, under approximate balancedness and deficiency margin initializations.

ABSTRACT

We analyze speed of convergence to global optimum for gradient descent training a deep linear neural network (parameterized as $x \mapsto W_N W_{N-1} \cdots W_1 x$) by minimizing the $\ell_2$ loss over whitened data. Convergence at a linear rate is guaranteed when the following hold: (i) dimensions of hidden layers are at least the minimum of the input and output dimensions; (ii) weight matrices at initialization are approximately balanced; and (iii) the initial loss is smaller than the loss of any rank-deficient solution. The assumptions on initialization (conditions (ii) and (iii)) are necessary, in the sense that violating any one of them may lead to convergence failure. Moreover, in the important case of output dimension 1, i.e. scalar regression, they are met, and thus convergence to global optimum holds, with constant probability under a random initialization scheme. Our results significantly extend previous analyses, e.g., of deep linear residual networks (Bartlett et al., 2018).

Motivation & Objective

• Motivate and analyze why gradient-based optimization can succeed for deep linear networks.
• Establish conditions under which gradient descent converges to the global minimum at a linear rate for arbitrary depth.
• Characterize initialization properties (balancedness and deficiency margin) that ensure convergence.
• Extend trajectory-based analysis beyond residual networks to general deep linear architectures.

Proposed method

• Model deep linear networks with end-to-end weight W_{1:N} = W_N ... W_1 and minimize the L^N loss over whitened data.
• Cast the training as minimizing the Frobenius distance to a target matrix Phi: L^1(W)=0.5||W- Phi||_F^2, where Phi = Lambda_{yx}.
• Introduce and formalize approximate balancedness (W_{j+1}^T W_{j+1} ≈ W_j W_j^T) and deficiency margin (distance to Phi bounded away from rank deficiency).
• Prove a trajectory-based descent lemma showing decrease in L^1(W_{1:N}) at every step when sigma_min(W_{1:N}) is bounded away from zero.
• Derive a linear-rate convergence theorem under explicit initialization conditions and a suitable learning rate, yielding an O(log(1/epsilon)) iteration bound.

Experimental results

Research questions

• RQ1 Under what initialization conditions does gradient descent on deep linear networks converge to the global minimum when trained with the 2-norm loss on whitened data?
• RQ2 How do hidden-layer dimensions, initialization balance, and deficiency margin affect convergence speed and guarantees across arbitrary network depth?
• RQ3 Can trajectory-based analysis extend convergence results beyond shallow or residual-linear settings to general deep linear architectures?
• RQ4 What is the probability, under random initialization, that scalar regression (output dimension 1) meets the required conditions for convergence?

Key findings

• Gradient descent converges to the global minimum at a linear rate if (i) hidden dimensions are at least min(input, output) dimensions, (ii) initialization yields approximately balanced weights, and (iii) the initial loss is smaller than any rank-deficient solution's loss.
• These initialization conditions are necessary; violating any can lead to convergence failure.
• For scalar regression (output dimension 1), the required initialization conditions are met with constant probability under common random near-zero initialization.
• The analysis generalizes previous results for deep linear residual networks to arbitrary depth and width configurations.
• A deficiency margin implies all points within a sublevel set have full-rank end-to-end mappings, strengthening convergence guarantees when combined with approximate balancedness.
• Theorem 1 provides an explicit iteration bound T ≥ (1/(η c^{2(N-1)/N})) log(ℓ(0)/ε) for achieving ε accuracy, with balance and margin ensuring descent.