[Paper Review] Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies

Sarath Chandar, Chinnadhurai Sankar+3 more|arXiv (Cornell University)|Jan 22, 2019

Advanced Neural Network Applications31 references4 citations

TL;DR

This paper proposes the Non-saturating Recurrent Unit (NRU), a novel recurrent architecture that replaces saturating activation functions and gates with non-saturating ReLU-based components to improve gradient flow. The NRU achieves state-of-the-art performance on long-term dependency tasks, outperforming LSTM, GRU, and other RNN variants across multiple benchmarks including permuted MNIST and memory copy tasks.

ABSTRACT

Modelling long-term dependencies is a challenge for recurrent neural networks. This is primarily due to the fact that gradients vanish during training, as the sequence length increases. Gradients can be attenuated by transition operators and are attenuated or dropped by activation functions. Canonical architectures like LSTM alleviate this issue by skipping information through a memory mechanism. We propose a new recurrent architecture (Non-saturating Recurrent Unit; NRU) that relies on a memory mechanism but forgoes both saturating activation functions and saturating gates, in order to further alleviate vanishing gradients. In a series of synthetic and real world tasks, we demonstrate that the proposed model is the only model that performs among the top 2 models across all tasks with and without long-term dependencies, when compared against a range of other architectures.

Motivation & Objective

• Address the vanishing gradient problem in recurrent neural networks that hinders learning of long-term dependencies.
• Overcome limitations of saturating activation functions (e.g., sigmoid, tanh) and saturating gates in LSTM and GRU, which attenuate gradients over time.
• Develop a new recurrent unit architecture that maintains stable gradient flow over long sequences by avoiding gradient saturation.
• Demonstrate that non-saturating components can outperform traditional gated RNNs on both long- and short-term dependency tasks.
• Establish a new baseline for general-purpose RNNs in sequential modeling by introducing a non-saturating, memory-augmented architecture.

Proposed method

• Propose a new recurrent unit (NRU) that uses ReLU activation functions instead of saturating nonlinearities like sigmoid or tanh.
• Implement an additive memory mechanism similar to LSTM and GRU, enabling information to bypass transition operators and reduce gradient attenuation.
• Replace saturating gates with non-saturating, learnable gating mechanisms that do not saturate at extreme values, preserving gradient magnitude.
• Train the NRU using standard backpropagation with gradient clipping to prevent exploding gradients, while relying on non-saturating components for stable training.
• Use a memory state update rule that combines input and hidden state via additive pathways, similar to GRU and LSTM, but with non-saturating nonlinearity.
• Evaluate the NRU on synthetic tasks (e.g., memory copy, sequential MNIST) and real-world benchmarks to compare gradient flow, convergence speed, and accuracy.

Experimental results

Research questions

• RQ1 Can replacing saturating activation functions and gates with non-saturating alternatives improve gradient flow in recurrent networks?
• RQ2 Does a non-saturating gated recurrent unit (NRU) outperform standard LSTM and GRU on long-term dependency tasks?
• RQ3 How does the NRU perform in terms of convergence speed and training stability compared to other RNN variants on long sequences?
• RQ4 Can the NRU maintain high performance across diverse tasks, including both long- and short-term dependency problems?
• RQ5 What is the impact of non-saturating components on gradient norms and memory retention in recurrent networks?

Key findings

• The NRU achieved the highest test accuracy of 95.38% on the permuted sequential MNIST task, outperforming all other models including EURNN (94.50%) and GRU (92.39%).
• NRU demonstrated superior gradient flow, with significantly higher gradient norms during early training compared to LSTM-Chrono and JANET, indicating better signal propagation.
• The model converged faster and more stably than RNN-orth and RNN-id, especially with larger memory sizes (e.g., 144 units), where convergence occurred nearly twice as fast as with 64 units.
• NRU maintained stable training even with long sequences (T = 1000–2000), showing no divergence despite gradient spikes, suggesting robustness to long-term dependencies.
• In the memory copy task with increasing sequence lengths (T = 100, 200, 500), NRU showed stable training and correct memory retention, indicating effective long-term information storage.
• NRU outperformed LSTM-chrono, which did not show consistent improvement over standard LSTM, suggesting that chrono-initialization does not always enhance performance when non-saturating components are available.