[Paper Review] Regularizing and Optimizing LSTM Language Models

Stephen Merity, Nitish Shirish Keskar+1 more|arXiv (Cornell University)|Aug 7, 2017

Topic ModelingComputer Science31 references468 citations

TL;DR

The paper introduces AWD-LSTM (weight-dropped LSTM) and NT-ASGD to regularize and optimize LSTM language models, achieving state-of-the-art perplexities on Penn Treebank and WikiText-2, with further gains from a neural cache on top.

ABSTRACT

Recurrent neural networks (RNNs), such as long short-term memory networks (LSTMs), serve as a fundamental building block for many sequence learning tasks, including machine translation, language modeling, and question answering. In this paper, we consider the specific problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models. We propose the weight-dropped LSTM which uses DropConnect on hidden-to-hidden weights as a form of recurrent regularization. Further, we introduce NT-ASGD, a variant of the averaged stochastic gradient method, wherein the averaging trigger is determined using a non-monotonic condition as opposed to being tuned by the user. Using these and other regularization strategies, we achieve state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2. In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2.

Motivation & Objective

• Motivate effective regularization for over-parameterized RNNs without modifying LSTM implementations.
• Propose weight-dropped LSTM (DropConnect on hidden-to-hidden weights) for recurrent regularization.
• Investigate optimization strategies, notably NT-ASGD, to improve training of regularized LSTMs.
• Explore extended regularization (variable-length BPTT, embedding dropout, AR/TAR, weight tying) to improve data efficiency and generalization.
• Evaluate on PTB and WT2 to establish state-of-the-art perplexities and assess neural cache gains.

Proposed method

• Introduce weight-dropped LSTM with DropConnect on recurrent weight matrices to regularize recurrent connections without modifying LSTM internals.
• Use NT-ASGD, a non-monotonic trigger variant of averaged SGD, with a fixed learning rate to improve training stability and performance.
• Apply variable-length backpropagation through time to utilize data more efficiently during training.
• Implement embedding dropout and variational dropout for different parts of the model.
• Employ embedding and softmax weight tying to reduce parameters and improve generalization.
• Incorporate activation regularization (AR) and temporal activation regularization (TAR) applied to the final LSTM layer outputs.

Experimental results

Research questions

• RQ1 Can recurrent regularization via DropConnect on hidden-to-hidden weights improve generalization in word-level language models without changing LSTM implementations?
• RQ2 Does NT-ASGD provide practical and performance gains over standard SGD/ASGD in training regularized LSTMs for language modeling?
• RQ3 What is the impact of extended regularization techniques (variable-length BPTT, embedding dropout, AR/TAR, weight tying) on perplexity on PTB and WT2?
• RQ4 How does a neural cache interact with AWD-LSTM to further reduce perplexity on PTB and WT2?

Key findings

• AWD-LSTM achieves state-of-the-art word-level perplexities on Penn Treebank (57.3) and WikiText-2 (65.8) with a vanilla LSTM.
• Applying a neural cache on top of AWD-LSTM yields further improvements to 52.8 (PTB) and 52.0 (WT2).
• NT-ASGD with a non-monotonic averaging trigger outperforms SGD-based training for these regularized LSTMs.
• Extended regularization techniques (variable-length BPTT, embedding dropout, AR/TAR, weight tying) contribute to substantial perplexity improvements and data efficiency.
• Weight-dropped LSTM (DropConnect on recurrent weights) is a critical element; removing it causes large perplexity increases (up to 11 points).
• Fine-tuning ASGD after NT-ASGD provides additional gains; removing this step degrades performance.