[Paper Review] Gradient descent aligns the layers of deep linear networks

Ziwei Ji, Matus Telgarsky|arXiv (Cornell University)|Oct 4, 2018

Sparse and Compressive Sensing TechniquesEngineering12 references22 citations

TL;DR

This paper demonstrates that gradient descent and gradient flow on deep linear networks with linearly separable data lead to risk convergence to zero and implicit regularization through layer alignment: each weight matrix asymptotically becomes rank-1 and adjacent layers align in direction, ultimately causing the network's overall linear predictor to converge to the maximum margin solution. This alignment is shown to emerge naturally from optimization dynamics without additional assumptions.

ABSTRACT

This paper establishes risk convergence and asymptotic weight matrix alignment --- a form of implicit regularization --- of gradient flow and gradient descent when applied to deep linear networks on linearly separable data. In more detail, for gradient flow applied to strictly decreasing loss functions (with similar results for gradient descent with particular decreasing step sizes): (i) the risk converges to 0; (ii) the normalized i-th weight matrix asymptotically equals its rank-1 approximation $u_iv_i^{ op}$; (iii) these rank-1 matrices are aligned across layers, meaning $|v_{i+1}^{ op}u_i| o1$. In the case of the logistic loss (binary cross entropy), more can be said: the linear function induced by the network --- the product of its weight matrices --- converges to the same direction as the maximum margin solution. This last property was identified in prior work, but only under assumptions on gradient descent which here are implied by the alignment phenomenon.

Motivation & Objective

• To understand the implicit bias of gradient descent in deep linear networks trained on linearly separable data.
• To establish that risk converges to zero under gradient flow and gradient descent with decreasing step sizes.
• To prove that weight matrices asymptotically become rank-1 and align across layers, implying implicit regularization.
• To show that the final network predictor converges to the maximum margin solution under logistic loss.
• To unify risk minimization and structural alignment in the optimization dynamics of deep linear networks.

Proposed method

• Analyzes gradient flow on strictly decreasing loss functions, proving risk convergence to zero and asymptotic rank-1 approximation of each weight matrix.
• Uses Frobenius norm normalization to show that the ratio of spectral to Frobenius norm of each weight matrix tends to 1, indicating rank-1 structure.
• Proves inter-layer alignment via the inner product |v_{i+1}^T u_i| → 1, where u_i and v_{i+1} are top singular vectors of adjacent layers.
• Demonstrates that for logistic loss, the product of weight matrices converges in direction to the maximum margin solution, aligning with the data's optimal separator.
• Employs differential equation analysis for gradient flow and discrete recurrence for gradient descent, with norm and projection dynamics to control orthogonal components.
• Uses orthogonal projection onto the span of the maximum margin direction to track convergence of weight matrices to the optimal subspace.

Experimental results

Research questions

• RQ1 Does gradient descent on deep linear networks with linearly separable data lead to risk convergence to zero?
• RQ2 Do individual weight matrices in deep linear networks asymptotically become rank-1 under gradient descent?
• RQ3 Is there alignment between the top singular vectors of adjacent layers during optimization?
• RQ4 Does the final network predictor converge to the maximum margin solution under logistic loss?
• RQ5 Can the implicit regularization of gradient descent be explained by structural alignment of layers without explicit regularization?

Key findings

• Risk converges to zero under gradient flow and gradient descent with decreasing step sizes on strictly decreasing loss functions.
• Each normalized weight matrix Wi / ||Wi||_F asymptotically equals its rank-1 approximation ui vi^T, implying ||Wi||_2 / ||Wi||_F → 1.
• Adjacent rank-1 approximations align: |v_{i+1}^T ui| → 1 across layers, indicating directional coherence.
• For the logistic loss, the final linear predictor w_prod = WL...W1 converges in direction to the maximum margin solution ū.
• The first-layer right singular vector v1 converges to the maximum margin direction ū, and the entire network aligns with it.
• The alignment phenomenon explains the implicit regularization of gradient descent, as orthogonal components are suppressed and norm is not wasted on inactive directions.