[Paper Review] Pointer Sentinel Mixture Models

Stephen Merity, Caiming Xiong+2 more|arXiv (Cornell University)|Sep 26, 2016

Natural Language Processing Techniques17 references481 citations

TL;DR

This paper introduces a hybrid pointer-sentinel mixture model that combines softmax vocabulary predictions with a pointer mechanism to copy from recent context, achieving state-of-the-art perplexity on Penn Treebank with far fewer parameters and introducing WikiText as a new longer-context language modeling benchmark.

ABSTRACT

Recent neural network sequence models with softmax classifiers have achieved their best language modeling performance only with very large hidden states and large vocabularies. Even then they struggle to predict rare or unseen words even if the context makes the prediction unambiguous. We introduce the pointer sentinel mixture architecture for neural sequence models which has the ability to either reproduce a word from the recent context or produce a word from a standard softmax classifier. Our pointer sentinel-LSTM model achieves state of the art language modeling performance on the Penn Treebank (70.9 perplexity) while using far fewer parameters than a standard softmax LSTM. In order to evaluate how well language models can exploit longer contexts and deal with more realistic vocabularies and larger corpora we also introduce the freely available WikiText corpus.

Motivation & Objective

• Motivate and address the difficulty of predicting rare or unseen words in language models with limited hidden state capacity.
• Propose a mixture architecture that can emit either a word from the softmax vocabulary or copy a word from recent context via a pointer network.
• Introduce a sentinel-based gating mechanism that decides when to use the pointer versus the softmax component.
• Share a new large-language modeling dataset (WikiText) to evaluate long-range dependencies and realistic vocabularies.
• Demonstrate state-of-the-art perplexity on Penn Treebank with reduced parameter count compared to comparable models.

Proposed method

• Define a softmax-RNN component for standard word prediction.
• Develop a pointer network component that uses attention over a window of past hidden states to copy words from the input.
• Create a pointer sentinel mixture that combines the pointer and softmax predictions via a gate g, with g derived from an augmented attention vector that includes a sentinel term.
• Introduce a sentinel value in the attention to allow the model to back off to the softmax when the pointer cannot confidently predict the next word.
• Train with cross-entropy loss adapted to the mixture, and regenerate stale RNN outputs within the pointer window during backpropagation through time.
• Evaluate on Penn Treebank and WikiText datasets, with detailed comparisons to prior LSTM-based language models.

Experimental results

Research questions

• RQ1 Can a hybrid model that combines a pointer mechanism with a traditional softmax output improve language modeling, especially for rare or OoV words?
• RQ2 Does integrating a sentinel-based gating mechanism enable effective sharing of parameters and improve decision making between pointer and softmax components?
• RQ3 How does the pointer sentinel mixture perform on longer-context datasets like WikiText compared to standard LSTMs?
• RQ4 What is the impact of the pointer window length L and training strategy (BPTT) on learning long-range dependencies?

Key findings

• The pointer sentinel-LSTM achieves state-of-the-art perplexity on Penn Treebank (70.9) using substantially fewer parameters than large LSTMs.
• On Penn Treebank, the medium pointer sentinel-LSTM outperforms large LSTM variants while using roughly one third the parameters.
• On WikiText-2, the pointer sentinel-LSTM significantly outperforms variational LSTM baselines (84.8 vs. 101.7 validation perplexity; 80.8 vs. 96.3 test perplexity).
• The model better handles rare words by enabling the pointer component to copy from recent context, with notable gains as word frequency decreases.
• The sentinel-based gating enables effective back-off to the softmax when the pointer cannot confidently match a word, improving robustness and performance.