[论文解读] A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

Sanjeev Arora, Nadav Cohen+2 more|arXiv (Cornell University)|Oct 4, 2018

Stochastic Gradient Optimization Techniques参考文献 39被引用 113

一句话总结

论文证明在白化数据上使用 2-范数损失训练的深层线性网络，梯度下降在近似平衡性和缺陷裕度初始化条件下可线性收敛至全局最小值。

ABSTRACT

We analyze speed of convergence to global optimum for gradient descent training a deep linear neural network (parameterized as $x \mapsto W_N W_{N-1} \cdots W_1 x$) by minimizing the $\ell_2$ loss over whitened data. Convergence at a linear rate is guaranteed when the following hold: (i) dimensions of hidden layers are at least the minimum of the input and output dimensions; (ii) weight matrices at initialization are approximately balanced; and (iii) the initial loss is smaller than the loss of any rank-deficient solution. The assumptions on initialization (conditions (ii) and (iii)) are necessary, in the sense that violating any one of them may lead to convergence failure. Moreover, in the important case of output dimension 1, i.e. scalar regression, they are met, and thus convergence to global optimum holds, with constant probability under a random initialization scheme. Our results significantly extend previous analyses, e.g., of deep linear residual networks (Bartlett et al., 2018).

研究动机与目标

• Motivate and analyze why gradient-based optimization can succeed for deep linear networks.
• Establish conditions under which gradient descent converges to the global minimum at a linear rate for arbitrary depth.
• Characterize initialization properties (balancedness and deficiency margin) that ensure convergence.
• Extend trajectory-based analysis beyond residual networks to general deep linear architectures.

提出的方法

• Model deep linear networks with end-to-end weight W_{1:N} = W_N ... W_1 and minimize the L^N loss over whitened data.
• Cast the training as minimizing the Frobenius distance to a target matrix Phi: L^1(W)=0.5||W- Phi||_F^2, where Phi = Lambda_{yx}.
• Introduce and formalize approximate balancedness (W_{j+1}^T W_{j+1} ≈ W_j W_j^T) and deficiency margin (distance to Phi bounded away from rank deficiency).
• Prove a trajectory-based descent lemma showing decrease in L^1(W_{1:N}) at every step when sigma_min(W_{1:N}) is bounded away from zero.
• Derive a linear-rate convergence theorem under explicit initialization conditions and a suitable learning rate, yielding an O(log(1/epsilon)) iteration bound.

实验结果

研究问题

• RQ1 Under what initialization conditions does gradient descent on deep linear networks converge to the global minimum when trained with the 2-norm loss on whitened data?
• RQ2 How do hidden-layer dimensions, initialization balance, and deficiency margin affect convergence speed and guarantees across arbitrary network depth?
• RQ3 Can trajectory-based analysis extend convergence results beyond shallow or residual-linear settings to general deep linear architectures?
• RQ4 What is the probability, under random initialization, that scalar regression (output dimension 1) meets the required conditions for convergence?

主要发现

• Gradient descent converges to the global minimum at a linear rate if (i) hidden dimensions are at least min(input, output) dimensions, (ii) initialization yields approximately balanced weights, and (iii) the initial loss is smaller than any rank-deficient solution's loss.
• These initialization conditions are necessary; violating any can lead to convergence failure.
• For scalar regression (output dimension 1), the required initialization conditions are met with constant probability under common random near-zero initialization.
• The analysis generalizes previous results for deep linear residual networks to arbitrary depth and width configurations.
• A deficiency margin implies all points within a sublevel set have full-rank end-to-end mappings, strengthening convergence guarantees when combined with approximate balancedness.
• Theorem 1 provides an explicit iteration bound T ≥ (1/(η c^{2(N-1)/N})) log(ℓ(0)/ε) for achieving ε accuracy, with balance and margin ensuring descent.