[Paper Review] On the Convergence Rate of Training Recurrent Neural Networks

Zeyuan Allen-Zhu, Yuanzhi Li+1 more|arXiv (Cornell University)|Oct 29, 2018

Advanced Neural Network Applications66 references47 citations

TL;DR

The paper proves that gradient descent and stochastic gradient descent can train Elman RNNs with ReLU to near-zero training error under over-parameterization, and provides new analysis tools for multi-layer networks.

ABSTRACT

How can local-search methods such as stochastic gradient descent (SGD) avoid bad local minima in training multi-layer neural networks? Why can they fit random labels even given non-convex and non-smooth architectures? Most existing theory only covers networks with one hidden layer, so can we go deeper? In this paper, we focus on recurrent neural networks (RNNs) which are multi-layer networks widely used in natural language processing. They are harder to analyze than feedforward neural networks, because the $ extit{same}$ recurrent unit is repeatedly applied across the entire time horizon of length $L$, which is analogous to feedforward networks of depth $L$. We show when the number of neurons is sufficiently large, meaning polynomial in the training data size and in $L$, then SGD is capable of minimizing the regression loss in the linear convergence rate. This gives theoretical evidence of how RNNs can memorize data. More importantly, in this paper we build general toolkits to analyze multi-layer networks with ReLU activations. For instance, we prove why ReLU activations can prevent exponential gradient explosion or vanishing, and build a perturbation theory to analyze first-order approximation of multi-layer networks.

Motivation & Objective

• Understand why ReLU stabilizes training to avoid vanishing/exploding gradients in RNNs.
• Show that RNNs can be trained efficiently to near-zero training error on non-degenerate data.
• Develop a general toolkit for analyzing multi-layer networks with ReLU activations.
• Demonstrate extension to deep RNNs and related loss functions under mild assumptions.
• Clarify how over-parameterization enables SGD to avoid bad local minima in multi-layer architectures.

Proposed method

• Analyze the classical Elman RNN with ReLU activations and train the hidden weight W while A,B are initialized randomly.
• Prove GD/SGD achieve f(W) ≤ ε with T = poly(n,d,L,δ^{-1}, log(1/ε)) iterations, starting from random Gaussian initializations.
• Introduce a Polyak-Łojasiewicz-type gradient lower bound (Theorem 3) and a semi-smoothness property (Theorem 4) to characterize first-order behavior.
• Develop forward/backward propagation analysis that controls gradient growth and avoids exponential explosion across L layers by leveraging over-parameterization and induced randomness.
• Provide robustness results under adversarial perturbations and discuss extension to deep RNNs and other Lipschitz-smooth losses.

Experimental results

Research questions

• RQ1 Can ReLU activations stabilize the training of recurrent neural networks to avoid vanishing/exploding gradients across long time horizons?
• RQ2 Is it possible to train multi-layer RNNs to near-zero training error efficiently using gradient-based methods under mild assumptions?
• RQ3 How does over-parameterization influence convergence and the avoidance of bad local minima in multi-layer RNNs?
• RQ4 Can the results be extended to deep RNNs and other Lipschitz-smooth loss functions?

Key findings

• Both GD and SGD converge to ε-training-error under over-parameterization with iteration count T = poly(n,d,L,δ^{-1}) · log(1/ε).
• The convergence rate is effectively linear in log(1/ε), despite non-convexity and multi-layer structure.
• A pair of technical results (gradient lower bound and semi-smoothness) underpin convergence, arguing against spurious local minima.
• ReLU activations help prevent exponential gradient explosion or vanishing across L layers.
• The results hold even when only the hidden weight W is trained while A,B are randomly initialized; data separability across layers is preserved through the network.