[Paper Review] Recovery Guarantees for One-hidden-layer Neural Networks

Kai Zhong, Zhao Song+3 more|arXiv (Cornell University)|Jun 10, 2017

Neural Networks and Applications19 references129 citations

TL;DR

The paper proves parameter recovery and global convergence guarantees for one-hidden-layer neural networks using Hessian analysis near the ground truth and a tensor-based initialization that yields linear-in-dimension sample and computation complexity under mild assumptions.

ABSTRACT

In this paper, we consider regression problems with one-hidden-layer neural networks (1NNs). We distill some properties of activation functions that lead to $\mathit{local~strong~convexity}$ in the neighborhood of the ground-truth parameters for the 1NN squared-loss objective. Most popular nonlinear activation functions satisfy the distilled properties, including rectified linear units (ReLUs), leaky ReLUs, squared ReLUs and sigmoids. For activation functions that are also smooth, we show $\mathit{local~linear~convergence}$ guarantees of gradient descent under a resampling rule. For homogeneous activations, we show tensor methods are able to initialize the parameters to fall into the local strong convexity region. As a result, tensor initialization followed by gradient descent is guaranteed to recover the ground truth with sample complexity $ d \cdot \log(1/ε) \cdot \mathrm{poly}(k,λ)$ and computational complexity $n\cdot d \cdot \mathrm{poly}(k,λ) $ for smooth homogeneous activations with high probability, where $d$ is the dimension of the input, $k$ ($k\leq d$) is the number of hidden nodes, $λ$ is a conditioning property of the ground-truth parameter matrix between the input layer and the hidden layer, $ε$ is the targeted precision and $n$ is the number of samples. To the best of our knowledge, this is the first work that provides recovery guarantees for 1NNs with both sample complexity and computational complexity $\mathit{linear}$ in the input dimension and $\mathit{logarithmic}$ in the precision.

Motivation & Objective

• Motivate understanding of regression with one-hidden-layer neural networks (1NNs) under Gaussian inputs.
• Identify conditions on activations that yield local strong convexity of the squared-loss near ground-truth parameters.
• Develop a tensor-based initialization to place parameters into the basin of attraction of the local strong convexity region.
• Establish a globally convergent training procedure with sample complexity linear in input dimension and logarithmic in precision.

Proposed method

• Characterize activation properties (Properties 3.1–3.3) that ensure positive definiteness of the Hessian near W*.
• Show local positive definiteness of the empirical Hessian and local linear convergence of gradient descent under these activation properties.
• Introduce tensor methods to initialize W and v to fall into the strong-convexity region (Algorithm 1).
• Reduce tensor-based initialization from cubic to linear dependence on dimension by first estimating a 2nd-order moment to recover a subspace V and then performing lower-dimensional tensor decomposition (P3(V,V,V)).
• Provide a globally convergent algorithm (Algorithm 2) combining tensor initialization with iterative gradient descent, with convergence guarantees (Theorem 6.1).

Experimental results

Research questions

• RQ1 Under what activation function conditions does the 1NN squared-loss admit local strong convexity near the ground-truth parameters?
• RQ2 Can tensor-based initialization place parameters into the basin of attraction to guarantee convergence of gradient-based methods?
• RQ3 What are the sample and computational complexities required to recover the ground-truth parameters for 1NNs under Gaussian inputs?
• RQ4 Does the proposed approach extend to smooth homogeneous activations with provable global convergence?

Key findings

• Activation properties lead to positive definiteness of the Hessian in a neighborhood of the ground truth with sufficiently many samples.
• For smooth homogeneous activations, gradient descent with resampling achieves linear convergence to the ground-truth parameters.
• Tensor initialization can recover weights and output weights with sample complexity and time complexity linear in the input dimension (up to poly factors).
• A globally convergent procedure combining tensor initialization and gradient descent recovers the ground-truth parameters with high probability, with sample complexity  ~ linear in d and log(1/epsilon).
• The work provides recovery guarantees for 1NNs with both sample complexity and computational complexity linear in d and logarithmic in epsilon under mild assumptions.