[Paper Review] Learning One-hidden-layer Neural Networks with Landscape Design

Rong Ge, Jason D. Lee+1 more|arXiv (Cornell University)|Nov 1, 2017

Stochastic Gradient Optimization Techniques26 references113 citations

TL;DR

The paper designs a non-convex objective with a favorable landscape for learning a one-hidden-layer network from Gaussian inputs, enabling SGD to converge to ground-truth parameters without over-parameterization.

ABSTRACT

We consider the problem of learning a one-hidden-layer neural network: we assume the input $x\in \mathbb{R}^d$ is from Gaussian distribution and the label $y = a^ op σ(Bx) + ξ$, where $a$ is a nonnegative vector in $\mathbb{R}^m$ with $m\le d$, $B\in \mathbb{R}^{m imes d}$ is a full-rank weight matrix, and $ξ$ is a noise vector. We first give an analytic formula for the population risk of the standard squared loss and demonstrate that it implicitly attempts to decompose a sequence of low-rank tensors simultaneously. Inspired by the formula, we design a non-convex objective function $G(\cdot)$ whose landscape is guaranteed to have the following properties: 1. All local minima of $G$ are also global minima. 2. All global minima of $G$ correspond to the ground truth parameters. 3. The value and gradient of $G$ can be estimated using samples. With these properties, stochastic gradient descent on $G$ provably converges to the global minimum and learn the ground-truth parameters. We also prove finite sample complexity result and validate the results by simulations.

Motivation & Objective

• Understand the population risk of a one-hidden-layer network with Gaussian inputs and ReLU-like activations.
• Design an objective with a nice optimization landscape where all local minima are global and correspond to ground-truth parameters.
• Demonstrate that SGD on the designed objective converges to the ground-truth parameters with finite-sample guarantees.

Proposed method

• Derive an analytic formula for the population risk of the standard squared loss and show its connection to simultaneous low-rank tensor decompositions (via Hermite coefficients).
• Propose a new non-convex objective G(B) that preserves ground-truth solutions and has no spurious local minima under certain conditions.
• In the orthogonal-B* setting, define G(B) with regularizers to enforce unit-norm rows and promote alignment with ground-truth components.
• Prove landscape properties of G: every local minimum corresponds to a permutation/sign-flip of ground-truth B*, and all saddles have negative curvature.
• Show that SGD on G converges to an approximate global minimum and provide a corollary for finite-sample settings (empirical G).
• Outline how a later procedure estimates a* once B* is recovered and then recovers the full parameters.

Experimental results

Research questions

• RQ1 Can a non-convex objective be crafted so that all local minima are global and correspond to ground-truth parameters for a one-hidden-layer network under Gaussian inputs?
• RQ2 What is the relationship between the population risk and tensor decomposition, and how can this guide landscape design?
• RQ3 Under which conditions (e.g., orthogonal B*, positive a*, regularization strength) does SGD converge to ground-truth parameters with finite-sample guarantees?
• RQ4 How can we estimate the ground-truth parameters from the designed objective without over-parameterization?
• RQ5 Does the empirical version of the designed objective preserve landscape guarantees with a polynomial number of samples?

Key findings

• A closed-form expression relates the population risk to a sum over Hermite coefficients times the Frobenius norm difference of tensor decompositions, linking learning to low-rank tensor decomposition.
• A new objective G(B) is constructed whose local minima are exactly permutations/sign-flips of the ground-truth B*, and whose saddles have negative curvature under suitable parameter choices.
• SGD on G(B) converges to a global minimum (up to permutation/sign) with polynomial-time guarantees in dimension and accuracy.
• When B* is orthogonal and suitable regularization is used, one can recover B* up to permutation/sign and then recover a* via linear regression, followed by standard parameter recovery steps.
• Finite-sample results show that the empirical version of G retains the landscape properties with a polynomial number of samples, enabling practical learning.