[Paper Review] Recurrent Neural Networks (RNNs): A gentle Introduction and Overview

Robin M. Schmidt|arXiv (Cornell University)|Nov 23, 2019

Topic ModelingComputer Science16 references150 citations

TL;DR

A concise overview of RNN fundamentals and advances, including BPTT, LSTMs, DRNNs, BRNNs, seq2seq, Attention, Transformer, and Pointer Networks.

ABSTRACT

State-of-the-art solutions in the areas of "Language Modelling & Generating Text", "Speech Recognition", "Generating Image Descriptions" or "Video Tagging" have been using Recurrent Neural Networks as the foundation for their approaches. Understanding the underlying concepts is therefore of tremendous importance if we want to keep up with recent or upcoming publications in those areas. In this work we give a short overview over some of the most important concepts in the realm of Recurrent Neural Networks which enables readers to easily understand the fundamentals such as but not limited to "Backpropagation through Time" or "Long Short-Term Memory Units" as well as some of the more recent advances like the "Attention Mechanism" or "Pointer Networks". We also give recommendations for further reading regarding more complex topics where it is necessary.

Motivation & Objective

• Introduce the fundamentals of Recurrent Neural Networks (RNNs) and their use in sequence data tasks.
• Explain training via Backpropagation Through Time (BPTT) and its truncated variant.
• Discuss key RNN architectures (LSTMs, DRNNs, BRNNs) and advanced models (Encoder-Decoder, seq2seq, Attention, Transformer, Pointer Networks).
• Provide guidance on reading and further topics through references to foundational papers.

Proposed method

• Present mathematical notation for hidden states and outputs in RNNs.
• Derive Backpropagation Through Time (BPTT) equations and highlight truncation to mitigate long-term dependencies.
• Describe LSTM gate computations and memory cell updates.
• Explain Bidirectional RNNs and how forward/backward states are combined for output.
• Introduce Encoder-Decoder (seq2seq) architecture and attention-based enhancements.
• Outline the Transformer architecture and the role of multi-head self-attention.
• Introduce Pointer Networks as a specialization of seq2seq with output pointers to input elements.

Experimental results

Research questions

• RQ1 What are the core computational steps and training procedures for basic RNNs and how do they differ from feedforward networks?
• RQ2 How do LSTMs and related gated architectures address vanishing/exploding gradient problems in long sequences?
• RQ3 What are the roles of encoder-decoder architectures, attention mechanisms, and Transformer models in sequence-to-sequence tasks?
• RQ4 How do Pointer Networks extend seq2seq to output pointers to input elements rather than fixed vocabulary outputs?

Key findings

• BPTT unfolds RNNs into a feedforward-like structure enabling gradient-based training across time steps.
• Truncated BPTT provides a practical upper bound on backpropagation through time to reduce computational load and stabilize training.
• LSTMs introduce gates and memory cells to mitigate vanishing gradients and enable learning over long sequences.
• Bidirectional RNNs incorporate future context by processing sequences in both directions and concatenating hidden states.
• Encoder-Decoder (seq2seq) models enable mapping from input sequences to output sequences, with attention improving alignment and performance.
• Transformer models replace recurrence with self-attention and enable parallelization, using multi-head attention and positional encoding; Pointer Networks further extend seq2seq by outputting input element pointers.