[Paper Review] Understanding the exploding gradient problem

Razvan Pascanu, Tomáš Mikolov+1 more|arXiv (Cornell University)|Nov 21, 2012

Model Reduction and Neural NetworksPhysics and Astronomy16 references405 citations

TL;DR

This paper investigates the root causes of the exploding gradient problem in training Recurrent Neural Networks (RNNs) through analytical, geometric, and dynamical systems perspectives. It proposes and empirically validates gradient norm clipping as an effective heuristic, demonstrating its necessity for achieving state-of-the-art performance on character prediction and polyphonic music generation tasks.

ABSTRACT

Training Recurrent Neural Networks is more troublesome than feedforward ones because of the vanishing and exploding gradient problems detailed in Bengio et al. (1994). In this paper we attempt to understand the fundamental issues underlying the exploding gradient problem by exploring it from an analytical, a geometric and a dynamical system perspective. Our analysis is used to justify the simple yet effective solution of norm clipping the exploded gradient. In the experimental section, the comparison between this heuristic solution and standard SGD provides empirical evidence towards our hypothesis as well as it shows that such a heuristic is required to reach state of the art results on a character prediction task and a polyphonic music prediction one.

Motivation & Objective

• To understand the fundamental causes of the exploding gradient problem in recurrent neural networks from multiple theoretical perspectives.
• To analyze the instability in gradient flow during backpropagation through time using analytical, geometric, and dynamical systems frameworks.
• To evaluate the effectiveness of gradient norm clipping as a practical solution to mitigate exploding gradients.
• To empirically demonstrate that norm clipping is essential for achieving state-of-the-art performance on sequence modeling benchmarks.

Proposed method

• Analytical investigation of the gradient computation in RNNs to identify conditions leading to exponential growth in gradients.
• Geometric analysis of the weight space to visualize how gradient paths diverge during training.
• Dynamical systems modeling to study the long-term behavior of gradient flow and identify instability thresholds.
• Application of gradient norm clipping as a heuristic to constrain gradient updates and stabilize training.
• Comparison of norm-clipped training with standard stochastic gradient descent (SGD) on benchmark sequence tasks.

Experimental results

Research questions

• RQ1 What are the underlying mathematical and dynamical mechanisms that cause gradients to explode in RNNs?
• RQ2 How do the geometric properties of the loss landscape contribute to gradient explosion?
• RQ3 To what extent does gradient norm clipping stabilize training and improve performance on sequence modeling tasks?
• RQ4 Is gradient norm clipping necessary to achieve state-of-the-art results in character and music sequence prediction?

Key findings

• The exploding gradient problem arises from the exponential accumulation of Jacobian matrices during backpropagation through time, leading to unstable training.
• Geometric analysis reveals that gradient vectors grow rapidly in magnitude due to unstable fixed points in the weight space.
• Norm clipping effectively stabilizes training by preventing extreme gradient updates, enabling convergence.
• Empirical results show that norm clipping is required to achieve state-of-the-art performance on both character prediction and polyphonic music generation tasks.