[Paper Review] Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN

Shuai Li, Wanqing Li+3 more|arXiv (Cornell University)|Mar 13, 2018

Neural Networks and Applications50 references125 citations

TL;DR

IndRNN introduces independent neurons within a layer, enabling training of much longer and deeper RNNs by regulating recurrent weights and using non-saturated activations like ReLU. It demonstrates superior long-sequence modeling and deep architectures over traditional RNNs and LSTMs.

ABSTRACT

Recurrent neural networks (RNNs) have been widely used for processing sequential data. However, RNNs are commonly difficult to train due to the well-known gradient vanishing and exploding problems and hard to learn long-term patterns. Long short-term memory (LSTM) and gated recurrent unit (GRU) were developed to address these problems, but the use of hyperbolic tangent and the sigmoid action functions results in gradient decay over layers. Consequently, construction of an efficiently trainable deep network is challenging. In addition, all the neurons in an RNN layer are entangled together and their behaviour is hard to interpret. To address these problems, a new type of RNN, referred to as independently recurrent neural network (IndRNN), is proposed in this paper, where neurons in the same layer are independent of each other and they are connected across layers. We have shown that an IndRNN can be easily regulated to prevent the gradient exploding and vanishing problems while allowing the network to learn long-term dependencies. Moreover, an IndRNN can work with non-saturated activation functions such as relu (rectified linear unit) and be still trained robustly. Multiple IndRNNs can be stacked to construct a network that is deeper than the existing RNNs. Experimental results have shown that the proposed IndRNN is able to process very long sequences (over 5000 time steps), can be used to construct very deep networks (21 layers used in the experiment) and still be trained robustly. Better performances have been achieved on various tasks by using IndRNNs compared with the traditional RNN and LSTM. The code is available at https://github.com/Sunnydreamrain/IndRNN_Theano_Lasagne.

Motivation & Objective

• Motivate the limitations of traditional RNNs in learning long-term dependencies due to gradient vanishing/exploding.
• Propose IndRNN where neurons in the same layer are independent and connected across layers to improve trainability and interpretability.
• Show that IndRNN can use non-saturated activations (e.g., ReLU) and support deep, residual architectures.
• Demonstrate through experiments that IndRNN outperforms traditional RNNs and LSTMs on tasks requiring long sequences and deep models.

Proposed method

• Define IndRNN with h_t = sigma(W x_t + u ⊙ h_{t-1} + b), where u is a vector of recurrent weights and ⊙ is the Hadamard product.
• Backpropagation through time for IndRNN yields gradients involving u^({T-t}) and sigma' terms, enabling explicit regulation of gradient flow.
• Derive memory retention bounds to prevent gradient vanishing and exploding by constraining |u_n| within a justified range.
• Demonstrate stacking multiple IndRNN layers (including residual connections) to construct very deep networks.
• Extend IndRNN to convolutional variants and integrate batch normalization and residual blocks for stability and performance.
• Provide interpretations of neuron behavior due to independence within layers.

Experimental results

Research questions

• RQ1 Can IndRNN maintain long-term dependencies beyond traditional RNNs and LSTMs?
• RQ2 Does making neurons independent within a layer and regulating recurrent weights enable training of much deeper and longer networks?
• RQ3 How do non-saturated activations (e.g., ReLU) impact gradient flow and training robustness in IndRNN?
• RQ4 What performance gains do IndRNNs achieve on tasks requiring long sequences, deep architectures, and various domains (language modeling, MNIST, action recognition)?

Key findings

• IndRNN can process very long sequences (over 5000 time steps).
• IndRNN enables very deep networks (up to 21 layers shown with language modeling).
• IndRNN with ReLU trains robustly and outperforms traditional RNNs and LSTM on several tasks.
• Two-layer IndRNN with independent neurons can represent traditional RNN behavior when appropriately arranged.
• Residual IndRNN architectures facilitate training of deeper networks and improve performance.
• IndRNN achieves strong results on sequential MNIST, character-level and word-level language modeling, and skeleton-based action recognition (NTU RGB+D).