[Paper Review] A Critical Review of Recurrent Neural Networks for Sequence Learning

Zachary C. Lipton, John Berkowitz+1 more|arXiv (Cornell University)|May 29, 2015

Multimodal Machine Learning Applications70 references2,087 citations

TL;DR

A survey of recurrent neural networks (RNNs) for sequence learning, covering architectures (like LSTM and BRNN), training challenges (vanishing/exploding gradients), and historical development, with emphasis on empirical results over biological plausibility.

ABSTRACT

Countless learning tasks require dealing with sequential data. Image captioning, speech synthesis, and music generation all require that a model produce outputs that are sequences. In other domains, such as time series prediction, video analysis, and musical information retrieval, a model must learn from inputs that are sequences. Interactive tasks, such as translating natural language, engaging in dialogue, and controlling a robot, often demand both capabilities. Recurrent neural networks (RNNs) are connectionist models that capture the dynamics of sequences via cycles in the network of nodes. Unlike standard feedforward neural networks, recurrent networks retain a state that can represent information from an arbitrarily long context window. Although recurrent neural networks have traditionally been difficult to train, and often contain millions of parameters, recent advances in network architectures, optimization techniques, and parallel computation have enabled successful large-scale learning with them. In recent years, systems based on long short-term memory (LSTM) and bidirectional (BRNN) architectures have demonstrated ground-breaking performance on tasks as varied as image captioning, language translation, and handwriting recognition. In this survey, we review and synthesize the research that over the past three decades first yielded and then made practical these powerful learning models. When appropriate, we reconcile conflicting notation and nomenclature. Our goal is to provide a self-contained explication of the state of the art together with a historical perspective and references to primary research.

Motivation & Objective

• Explain why modeling sequentiality explicitly is valuable for real-world tasks.
• Discuss limitations of Markov models and traditional feedforward nets in sequential settings.
• Provide a coherent, self-contained overview of RNN architectures, training challenges, and practical results.

Proposed method

• Review and synthesize three decades of RNN research.
• Clarify notation and unify terminology across conflicting sources.
• Explain forward and backward passes for RNNs and the role of backpropagation through time.

Experimental results

Research questions

• RQ1 Why is explicit sequential modeling necessary for practical tasks and long-range dependencies?
• RQ2 How do RNNs compare to Markov models in handling time dependencies and long-range context?
• RQ3 What architectures, training techniques, and optimizations have enabled successful large-scale RNN learning?
• RQ4 What are the historical milestones and key empirical findings in the development of RNNs?
• RQ5 How do modern RNN variants (like LSTM and BRNN) address training challenges and improve performance?

Key findings

• RNNs can capture long-range dependencies beyond fixed-context windows, addressing limitations of simple windows and Markov models.
• Training challenges such as vanishing and exploding gradients motivated the development of LSTM and related architectures.
• Backpropagation through time enables end-to-end training of RNNs across multiple time steps.
• BRNNs and LSTM architectures, along with advances in optimization and parallel computing, have driven substantial empirical progress on sequence tasks.
• Extensions like neural Turing machines (NTMs) and external memories further expand RNN capabilities.