[Paper Review] On the difficulty of training Recurrent Neural Networks

Razvan Pascanu, Tomáš Mikolov+1 more|arXiv (Cornell University)|Nov 21, 2012

Neural Networks and Applications23 references3,783 citations

TL;DR

The paper analyzes vanishing and exploding gradients in RNNs from analytical, geometric, and dynamical perspectives, and proposes gradient clipping plus a vanishing-gradient regularizer to improve training on long-range dependencies; it validates the approach empirically on synthetic tasks and real datasets.

ABSTRACT

There are two widely known issues with properly training Recurrent Neural Networks, the vanishing and the exploding gradient problems detailed in Bengio et al. (1994). In this paper we attempt to improve the understanding of the underlying issues by exploring these problems from an analytical, a geometric and a dynamical systems perspective. Our analysis is used to justify a simple yet effective solution. We propose a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem. We validate empirically our hypothesis and proposed solutions in the experimental section.

Motivation & Objective

• Investigate the causes of vanishing and exploding gradients in recurrent neural networks.
• Propose practical methods to mitigate exploding gradients via gradient norm clipping.
• Propose a soft vanishing-gradient regularization to maintain informative backpropagation through time.
• Empirically validate the proposed methods on synthetic tasks and real-world sequence modeling benchmarks.

Proposed method

• Derive gradient expressions to highlight exploding gradients using a sum-of-products form of backpropagation through time.
• Characterize conditions for gradient explosion via Jacobian products and spectral radius of the recurrent weight matrix.
• Propose gradient norm clipping to cap large gradient norms during training.
• Introduce a vanishing-gradient regularizer that favors maintaining gradient norms when backpropagating through time.
• Use Theano to compute gradients and validate on synthetic and real datasets.

Experimental results

Research questions

• RQ1 Do exploding gradients occur in RNNs with long-term dependencies, and under what conditions?
• RQ2 Can gradient norm clipping stabilize training and allow learning of long-range correlations?
• RQ3 Does a soft vanishing-gradient regularizer improve learning of long-term dependencies without harming short-term performance?
• RQ4 How do the proposed methods perform on both synthetic pathological tasks and real-world sequence modeling datasets?
• RQ5 How does the approach compare to existing strategies (e.g., LSTM, Hessian-free) in terms of performance and generalization?

Key findings

• Gradient clipping (norm-based) effectively controls exploding gradients and improves training stability.
• A soft vanishing-gradient regularizer can help preserve useful temporal dependencies without forcing strict equality of gradient flows.
• SGD with clipping and regularization (SGD-CR) solves long-sequence tasks that require memory, including the temporal order problem up to length 200.
• On polyphonic music prediction and language modeling, SGD-CR improves or matches state-of-the-art results across several datasets.
• Clipping and regularization yield strong empirical gains on both synthetic pathological problems and real-world tasks, with improved generalization.