[Paper Review] An Analysis of Neural Language Modeling at Multiple Scales

Stephen Merity, Nitish Shirish Keskar+1 more|arXiv (Cornell University)|Mar 22, 2018

Topic Modeling29 references143 citations

TL;DR

The paper demonstrates that well-tuned LSTM and QRNN baselines with tied adaptive softmax can achieve state-of-the-art results on both character-level and word-level language modeling across multiple scales, training efficiently on a single modern GPU.

ABSTRACT

Many of the leading approaches in language modeling introduce novel, complex and specialized architectures. We take existing state-of-the-art word level language models based on LSTMs and QRNNs and extend them to both larger vocabularies as well as character-level granularity. When properly tuned, LSTMs and QRNNs achieve state-of-the-art results on character-level (Penn Treebank, enwik8) and word-level (WikiText-103) datasets, respectively. Results are obtained in only 12 hours (WikiText-103) to 2 days (enwik8) using a single modern GPU.

Motivation & Objective

• Motivate that well-tuned baseline models can rival complex architectures on large-scale language modeling tasks.
• Extend state-of-the-art word-level models to larger vocabularies and character-level data.
• Analyze the LSTM versus QRNN trade-offs for character- and word-level tasks.
• Investigate hyperparameter importance to guide practical model tuning.
• Discuss dataset choices and evaluation metrics for fair comparisons.

Proposed method

• Use a baseline model framework with a trainable embedding layer, stacked recurrent layers, and a softmax classifier with tied weights.
• Compare LSTM and QRNN cells, highlighting GPU utilization and training speed benefits of QRNN.
• Employ longer truncated BPTT lengths to capture long-range dependencies and improve GPU efficiency.
• Adopt a modified adaptive softmax with weight tying to handle large vocabularies efficiently.
• Train and evaluate on three datasets: Penn Treebank (character), enwik8 (character), and WikiText-103 (word-level).
• Perform hyperparameter analyses, including random forest-based importance assessment and sensitivity studies.

Experimental results

Research questions

• RQ1 Can a standard LSTM/QRNN baseline achieve state-of-the-art results on both character- and word-level language modeling at scale?
• RQ2 How do LSTM and QRNN compare on character-level versus word-level tasks, and what hyperparameters most influence performance?
• RQ3 What is the impact of longer BPTT and tied adaptive softmax on training speed and model accuracy for large vocabularies?
• RQ4 Are commonly used benchmarks (e.g., Penn Treebank) suitable for character-level modeling, and how do dataset choices affect results?
• RQ5 How should practitioners tune hyperparameters to achieve strong performance with minimal tuning resources?

Key findings

• Well-tuned LSTM and QRNN baselines achieve competitive or state-of-the-art perplexities/BPC on the evaluated datasets.
• QRNN offers substantial speed advantages over LSTM on word-level WikiText-103 (training time and batch efficiency), but can underperform on realistic character-level tasks when dataset complexity increases.
• Longer BPTT and the use of tied adaptive softmax help manage large vocabularies effectively without resorting to overly complex architectures.
• Hyperparameter importance analyses indicate dropout-related hyperparameters (weight, embedding, and hidden dropouts) have strong impact on performance; embedding size is comparatively less influential within reasonable ranges.
• Penn Treebank character-level data is deemed flawed for character modeling, highlighting the need for more realistic datasets like enwik8 for evaluation.
• Parameter count is not a reliable proxy for model complexity or hardware requirements; practical efficiency depends on architecture and training dynamics.