[Paper Review] Exploring the Limits of Language Modeling

Rafał Józefowicz, Oriol Vinyals+3 more|arXiv (Cornell University)|Feb 7, 2016

Topic Modeling51 references915 citations

TL;DR

The paper trains very large LSTM-based language models on the One Billion Word Benchmark, introducing CNN-based Softmax and character-level embeddings to drastically reduce parameters and perplexity, and achieving state-of-the-art results with single models and ensembles.

ABSTRACT

In this work we explore recent advances in Recurrent Neural Networks for large scale Language Modeling, a task central to language understanding. We extend current models to deal with two key challenges present in this task: corpora and vocabulary sizes, and complex, long term structure of language. We perform an exhaustive study on techniques such as character Convolutional Neural Networks or Long-Short Term Memory, on the One Billion Word Benchmark. Our best single model significantly improves state-of-the-art perplexity from 51.3 down to 30.0 (whilst reducing the number of parameters by a factor of 20), while an ensemble of models sets a new record by improving perplexity from 41.0 down to 23.7. We also release these models for the NLP and ML community to study and improve upon.

Motivation & Objective

• Motivate and study language modeling at large scale beyond PTB using the One Billion Word Benchmark.
• Improve perplexity with reduced parameter counts via architectural innovations (LSTMs with projections, dropout, CNN-based embeddings).
• Investigate training-time efficiency and data efficiency through sampling-based losses and regularization strategies.
• Provide open-source models and training recipes to accelerate research in large-scale LMs.

Proposed method

• Use large-scale recurrent neural networks (LSTMs) with projection/bottleneck layers.
• Introduce CNN-based word embeddings and a CNN-based Softmax (eW = CNN(chars_w)) to reduce parameter count.
• Employ an importance sampling (IS) loss as an efficient alternative to full Softmax and relate it to Noise Contrastive Estimation (NCE).
• Combine word-level models with a character-level LSTM predictor (Char LSTM) as a hybrid to handle long-tail and OOV words.
• Train with dropout on non-recurrent connections, large-batch asynchronous GPU training, and AdaGrad optimization.
• Experiment with data from the 1B Word Benchmark and release models/recipes.

Experimental results

Research questions

• RQ1 Can LSTM-based language models be scaled to a very large vocabulary and dataset while maintaining or improving perplexity?
• RQ2 Do character-level embeddings and CNN-based Softmax reduce parameter counts without sacrificing performance?
• RQ3 How do IS and NCE losses compare for training large-scale LMs, and is IS data-efficient?
• RQ4 What is the impact of model size, regularization, and ensemble methods on perplexity and tail-word performance?
• RQ5 Can CNN-based and hybrid word-character approaches better handle OOV words and multilingual/morphologically rich text?

Key findings

• Single best model achieves perplexity 30.0 with 1.04B parameters (BIG LSTM+CNN Inputs).
• CNN Softmax with 128-dim correction improves performance, achieving 35.8 perplexity with 0.39B parameters.
• Character CNN embeddings reduce input-layer parameters by about 11x (72M vs 820M) while maintaining competitive performance.
• IS loss outperforms NCE in training speed and final perplexity for large-scale LMs.
• Ensemble of 10 LSTMs with supplemental components reaches 23.7 perplexity, a large improvement over prior work.
• Tail words are better modeled by LSTMs than KN-5 as shown in log-probability analysis.