[Paper Review] An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis

Yuandong Tian|arXiv (Cornell University)|Mar 2, 2017

Complex Network Analysis Techniques16 references78 citations

TL;DR

The paper derives a closed-form population gradient for a two-layer ReLU network under Gaussian inputs, and uses it to analyze critical points and convergence, including spontaneous symmetry breaking. It provides conditions under which gradient descent converges to the teacher weights and characterizes out-of-plane critical points as non-isolated manifolds.

ABSTRACT

In this paper, we explore theoretical properties of training a two-layered ReLU network $g(\mathbf{x}; \mathbf{w}) = \sum_{j=1}^K σ(\mathbf{w}_j^T\mathbf{x})$ with centered $d$-dimensional spherical Gaussian input $\mathbf{x}$ ($σ$=ReLU). We train our network with gradient descent on $\mathbf{w}$ to mimic the output of a teacher network with the same architecture and fixed parameters $\mathbf{w}^*$. We show that its population gradient has an analytical formula, leading to interesting theoretical analysis of critical points and convergence behaviors. First, we prove that critical points outside the hyperplane spanned by the teacher parameters ("out-of-plane") are not isolated and form manifolds, and characterize in-plane critical-point-free regions for two ReLU case. On the other hand, convergence to $\mathbf{w}^*$ for one ReLU node is guaranteed with at least $(1-ε)/2$ probability, if weights are initialized randomly with standard deviation upper-bounded by $O(ε/\sqrt{d})$, consistent with empirical practice. For network with many ReLU nodes, we prove that an infinitesimal perturbation of weight initialization results in convergence towards $\mathbf{w}^*$ (or its permutation), a phenomenon known as spontaneous symmetric-breaking (SSB) in physics. We assume no independence of ReLU activations. Simulation verifies our findings.

Motivation & Objective

• Develop a closed-form analytical expression for the population gradient of a two-layer ReLU network with Gaussian inputs.
• Characterize critical points, distinguishing in-plane and out-of-plane cases, and identify regions free of critical points.
• Analyze convergence of gradient descent to the teacher network under single and multiple ReLU nodes using Lyapunov methods.
• Demonstrate phenomena such as spontaneous symmetry breaking and its implications for initialization and convergence.
• Provide simulation validation of the theoretical results.

Proposed method

• Define the two-layer ReLU model g(x; w) = sum_j ReLU(w_j^T x) with teacher weights w* and centered spherical Gaussian input x.
• Derive the population gradient E[∇J(w)] for the L2 loss with Gaussian inputs, introducing the Population Gating (PG) function F(e, w).
• Obtain closed-form expression E[F(e, w)] = (N/2π)[(π−θ)w + ||w|| sin θ e], where θ is the angle between e and w.
• Show E[∇J] = E[F(w/||w||, w)] − E[F(w/||w||, w*)] and analyze its implications for learning dynamics.
• Develop normal equations for critical points in the K-ReLU setting as YE^T = B* W*^T, and study in-plane vs out-of-plane cases.
• Apply Lyapunov/LaSalle methods to establish convergence results for single ReLU and discuss symmetry-breaking in multi-ReLU settings.
• Extend the framework conceptually to multilayer ReLU networks via a proposition on the gradient structure (Eq. 19).

Experimental results

Research questions

• RQ1 What is the explicit form of the population gradient for a two-layer ReLU network with Gaussian inputs?
• RQ2 Where are the critical points located (in-plane vs out-of-plane) and can they be isolated?
• RQ3 Under what initialization conditions does gradient descent converge to the teacher weights for single and multiple ReLU nodes?
• RQ4 How does symmetry breaking manifest in multi-ReLU networks, and what are its implications for convergence?
• RQ5 Can the analytical framework extend to more complex (multilayer) architectures?

Key findings

• The population gradient has a closed-form decomposition into a linear-like term and a nonlinear term dependent on the angle between w and w*, enabling precise critical-point analysis.
• Out-of-plane critical points are non-isolated and form manifolds when d ≥ K+2, due to rotational symmetry around the principal hyperplane.
• For a single ReLU node, gradient descent converges to w* with high probability from random initialization with small enough variance, consistent with standard initialization practices.
• In the multi-ReLU case with orthonormal teacher weights, symmetric initialization leads to a saddle point, while infinitesimal perturbations induce convergence to w* or its permutation (spontaneous symmetry breaking).
• Simulations validate the analytical formula and demonstrate convergence trajectories, saddle points, and the effect of initialization on convergence behavior.