[Paper Review] Recurrent Neural Network Regularization

Wojciech Zaremba, Ilya Sutskever+1 more|arXiv (Cornell University)|Sep 8, 2014

Neural Networks and Applications32 references2,276 citations

TL;DR

The paper introduces a dropout-based regularization method for LSTMs that applies dropout only to non-recurrent connections, significantly reducing overfitting across language modeling, speech recognition, translation, and image captioning tasks.

ABSTRACT

We present a simple regularization technique for Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units. Dropout, the most successful technique for regularizing neural networks, does not work well with RNNs and LSTMs. In this paper, we show how to correctly apply dropout to LSTMs, and show that it substantially reduces overfitting on a variety of tasks. These tasks include language modeling, speech recognition, image caption generation, and machine translation.

Motivation & Objective

• Motivate the need for regularization in recurrent networks to prevent overfitting.
• Propose a dropout scheme that avoids harming LSTM memorization by not applying dropout to recurrent connections.
• Demonstrate the effectiveness of the method across multiple domains (language modeling, speech recognition, machine translation, image caption generation).
• Provide practical guidelines (dropout probabilities) and report empirical gains over non-regularized baselines.

Proposed method

• Apply dropout to non-recurrent connections only within LSTM-based RNNs.
• Use a dropout operator D on the input-to-hidden path while leaving recurrent connections intact.
• Describe the LSTM dynamics with standard gates (i, f, o, g) and show how dropout is integrated in the input path: [i f o g] = [sigm sigm sigm tanh] T_{2n,4n} [D(h^{l-1}_{t}); h^{l}_{t-1}].
• Maintain long-term memory by not applying dropout on recurrent connections, ensuring information can flow across many timesteps.
• Provide two experimental configurations: medium (650 units per layer, 50% dropout) and large (1500 units per layer, 65% dropout).
• Report training details (epochs, learning rate schedules, gradient clipping) and compare with non-regularized baselines.

Experimental results

Research questions

• RQ1 Does applying dropout only to non-recurrent connections preserve LSTM memorization while providing regularization benefits?
• RQ2 How does LSTM dropout affect performance across language modeling, speech recognition, machine translation, and image caption generation?
• RQ3 What dropout rates yield the best generalization for medium and large LSTM configurations?

Key findings

• In language modeling on PTB, medium regularized LSTM achieves 86.2 (val) and 82.7 (test) perplexity; large regularized LSTM achieves 82.2 (val) and 78.4 (test).
• +14–18 point improvements over non-regularized baselines depending on model size.
• In speech recognition (Icelandic dataset), regularized LSTM improves validation frame accuracy to 70.5 from 68.9 (training accuracy drops to 69.4).
• In machine translation (English→French, WMT’14), regularized LSTM attains perplexity 5.0 and BLEU 29.03, outperforming the non-regularized LSTM (perplexity 5.8, BLEU 25.9).
• In image captioning (MSCOCO), regularized model yields perplexity 7.99 and BLEU 24.3, vs non-regularized 8.47 and 23.5; an ensemble of 10 non-regularized models yields comparable gains (BLEU 24.4).
• Overall, dropping dropout on recurrent connections provides substantial regularization benefits across diverse RNN tasks without sacrificing the LSTM’s memory capabilities.