[Paper Review] Deep Independently Recurrent Neural Network (IndRNN)

Shuai Li, Wanqing Li+2 more|arXiv (Cornell University)|Oct 11, 2019

Neural Networks and Applications78 references35 citations

TL;DR

IndRNN introduces independently recurrent weights via a Hadamard product, enabling very deep RNNs and robust training with ReLU for long sequences. It outperforms traditional RNNs, LSTM, and Transformer on several tasks and is significantly faster.

ABSTRACT

Recurrent neural networks (RNNs) are known to be difficult to train due to the gradient vanishing and exploding problems and thus difficult to learn long-term patterns and construct deep networks. To address these problems, this paper proposes a new type of RNNs with the recurrent connection formulated as Hadamard product, referred to as independently recurrent neural network (IndRNN), where neurons in the same layer are independent of each other and connected across layers. Due to the better behaved gradient backpropagation, IndRNN with regulated recurrent weights effectively addresses the gradient vanishing and exploding problems and thus long-term dependencies can be learned. Moreover, an IndRNN can work with non-saturated activation functions such as ReLU (rectified linear unit) and be still trained robustly. Different deeper IndRNN architectures, including the basic stacked IndRNN, residual IndRNN and densely connected IndRNN, have been investigated, all of which can be much deeper than the existing RNNs. Furthermore, IndRNN reduces the computation at each time step and can be over 10 times faster than the commonly used Long short-term memory (LSTM). Experimental results have shown that the proposed IndRNN is able to process very long sequences and construct very deep networks. Better performance has been achieved on various tasks with IndRNNs compared with the traditional RNN, LSTM and the popular Transformer.

Motivation & Objective

• Address gradient vanishing and exploding in RNNs to learn long-term dependencies.
• Enable very deep RNN architectures with stable training.
• Allow use of non-saturated activations like ReLU while maintaining training robustness.
• Provide interpretability by making neurons in a layer independent.
• Improve computational efficiency to outperform traditional RNNs and LSTM in speed.

Proposed method

• Propose IndRNN where the recurrent input is u ⊙ h_{t-1} with a vector u, giving h_t = σ(W x_t + u ⊙ h_{t-1} + b).
• Extend IndRNN to convolutional variants and deep architectures (stacked, residual, densely connected).
• Derive backpropagation through time for IndRNN, showing gradient depends on u^T−t and σ′, enabling explicit gradient regulation.
• Establish memory–recurrent weight relationships and initialization ranges to control long-, short-, and no-memory dynamics.
• Demonstrate reductions in parameters and computation, with updates compatible with ReLU, batch norm, and dropout.
• Provide a CUDA-optimized PyTorch implementation and compare against LSTM, SRU, and Transformer on multiple tasks.

Experimental results

Research questions

• RQ1 Can IndRNN process very long sequences (e.g., thousands of steps) while maintaining stable gradients?
• RQ2 Can IndRNN be trained robustly with non-saturated activations such as ReLU?
• RQ3 How does IndRNN perform as a deep architecture with residual and densely connected designs?
• RQ4 What are the computational and parameter efficiency benefits of IndRNN compared with LSTM and Transformer?
• RQ5 How do IndRNN variants (basic, residual, dense) compare across tasks like language modeling, sequence classification, and action recognition?

Key findings

• IndRNN can process sequences over 5000 steps and supports very deep architectures (e.g., 21-layer residual IndRNN).
• Regulating the recurrent weights u_n controls gradient explosion/vanishing, enabling stable training with ReLU.
• IndRNN reduces computation per time step via element-wise recurrent operations, achieving over 10x speedups vs cuDNN LSTM in reported setups.
• IndRNN architecture variants (stacked, residual, densely connected) improve performance on sequential MNIST, language modeling, and skeleton-based action recognition.
• IndRNN with ReLU and proper regulation competes favorably with traditional RNNs, LSTM, and Transformer on multiple benchmarks.
• A CUDA-optimized, open-source implementation is provided for faster training compared with LSTM-based models.