[Paper Review] Overcoming the vanishing gradient problem in plain recurrent networks

Yuhuang Hu, Adrian E. G. Huber+2 more|arXiv (Cornell University)|Jan 18, 2018

Topic Modeling26 references83 citations

TL;DR

The paper introduces Recurrent Identity Networks (RINs), a gate-free plain RNN variant with a non-trainable identity surrogate memory that mitigates vanishing gradients and enables fast training, achieving competitive results with IRNNs and LSTMs on long-sequence tasks.

ABSTRACT

Plain recurrent networks greatly suffer from the vanishing gradient problem while Gated Neural Networks (GNNs) such as Long-short Term Memory (LSTM) and Gated Recurrent Unit (GRU) deliver promising results in many sequence learning tasks through sophisticated network designs. This paper shows how we can address this problem in a plain recurrent network by analyzing the gating mechanisms in GNNs. We propose a novel network called the Recurrent Identity Network (RIN) which allows a plain recurrent network to overcome the vanishing gradient problem while training very deep models without the use of gates. We compare this model with IRNNs and LSTMs on multiple sequence modeling benchmarks. The RINs demonstrate competitive performance and converge faster in all tasks. Notably, small RIN models produce 12%--67% higher accuracy on the Sequential and Permuted MNIST datasets and reach state-of-the-art performance on the bAbI question answering dataset.

Motivation & Objective

• Motivate the study of vanishing gradient in plain recurrent networks and limitations of gated architectures (LSTM/GRU).
• Explore whether plain RNNs can overcome vanishing gradients by leveraging a different memory mechanism.
• Propose and formalize the Recurrent Identity Network (RIN) as a gate-free model with a surrogate memory.
• Evaluate RIN against IRNN and LSTM on long-range sequence tasks (adding problem, MNIST variants, bAbI) to assess convergence and performance.

Proposed method

• Adopt an Iterative Estimation interpretation of RNNs to analyze information flow and identity mapping.
• Analyze dual gate designs in GNNs and show how carry gates enable identity-like behavior that can be emulated without gates.
• Propose RIN with the update h_t = ReLU(Wx_t + Uh_{t-1} + h_{t-1} + b) = ReLU(Wx_t + (U + I)h_{t-1} + b).
• Introduce a non-trainable identity matrix as surrogate memory to preserve past representations across steps.
• Discuss connections to Differential State Framework (DSF) and how RIN supports stable gradient propagation with ReLU activations.

Experimental results

Research questions

• RQ1 Can a plain recurrent network trained with ReLU mitigate vanishing gradients without gating mechanisms?
• RQ2 Does incorporating a non-trainable identity component preserve past representations and enable long-range dependency modeling?
• RQ3 How does RIN perform relative to IRNN and LSTM on long sequence tasks like adding problems, Sequential/Permuted MNIST, and bAbI tasks?

Key findings

• RINs converge faster than IRNNs and LSTMs in early training stages across tasks.
• Small RINs achieve 12%–67% higher accuracy on Sequential and Permuted MNIST compared to IRNNs and LSTMs in some configurations.
• RINs reach state-of-the-art performance on the bAbI question answering dataset in the reported setup.
• LSTMs generally perform worse than RINs and IRNNs on the MNIST tasks in the reported experiments.
• RINs exhibit competitive performance with gate-based models while using a gate-free design and a simple surrogate memory.