[Paper Review] Recent Advances in Recurrent Neural Networks

Hojjat Salehinejad, Sharan Sankar+3 more|arXiv (Cornell University)|Dec 29, 2017

Time Series Analysis and Forecasting96 references580 citations

TL;DR

A survey of recurrent neural networks (RNNs) covering fundamentals, training challenges like vanishing/exploding gradients, and a survey of architectures and optimization methods including BPTT, LSTM variants, and HF/Adam EKF approaches.

ABSTRACT

Recurrent neural networks (RNNs) are capable of learning features and long term dependencies from sequential and time-series data. The RNNs have a stack of non-linear units where at least one connection between units forms a directed cycle. A well-trained RNN can model any dynamical system; however, training RNNs is mostly plagued by issues in learning long-term dependencies. In this paper, we present a survey on RNNs and several new advances for newcomers and professionals in the field. The fundamentals and recent advances are explained and the research challenges are introduced.

Motivation & Objective

• Explain the fundamentals of discrete-time RNNs and their memory through hidden states.
• Review training challenges such as vanishing and exploding gradients and their causes.
• Survey major RNN architectures and regularization approaches in time-series modeling.
• Highlight optimization and training methods including gradient-based, Kalman-filter-based, second-order, and global search strategies.

Proposed method

• Present the standard RNN formulation with input, hidden, and output layers and equations for state update and output.
• Discuss activation functions and their impact on gradient flow and learning.
• Survey training methods including Back-propagation Through Time (BPTT), gradient clipping, and optimization algorithms (SGD, Adam, momentum, Nesterov).
• Describe Extended Kalman Filter-based learning and its decoupled variant for online training.
• Explain Hessian-Free optimization and its role in training deep or recurrent models.
• Cover global optimization approaches and evolutionary methods for RNN topology and parameter search.

Experimental results

Research questions

• RQ1 What are the core challenges in training RNNs for long-term dependencies?
• RQ2 How do different gradient-based and non-gradient-based methods address vanishing and exploding gradients in RNNs?
• RQ3 What are the effects of various RNN architectures and training regimes on performance in time-series tasks?
• RQ4 What optimization frameworks (second-order, HF, Kalman-based, global) are viable for RNN training and when are they advantageous?

Key findings

• RNNs can model dynamic systems but training is hindered by vanishing and exploding gradients.
• Back-propagation through time expands gradients across time, illustrating long-term dependency challenges.
• A breadth of training methods exists, including SGD, Adam, momentum, gradient clipping, EKF/DEKF, and Hessian-Free optimization.
• Hybrid architectures with deep transitions or deep outputs can improve performance in certain tasks.
• Global optimization and evolutionary strategies offer alternative avenues for topology and parameter tuning, despite higher complexity.
• HF optimization and online EKF-based methods show promise for training RNNs on non-stationary or online data.