[Paper Review] Evaluating Distributional Distortion in Neural Language Modeling

Benjamin LeBrun, Alessandro Sordoni+1 more|arXiv (Cornell University)|Mar 24, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces a controlled evaluation framework to assess how neural language models (LSTMs and Transformers) estimate probabilities for rare, novel sequences in natural language. By using a pre-trained Transformer as a ground-truth generative model to create artificial languages with exact sequence probabilities, the study reveals that neural LMs systematically underestimate the likelihood of well-formed rare sequences while overestimating ill-formed ones, with underestimation persisting even with more training data and worsening in lower-entropy distributions.

ABSTRACT

A fundamental characteristic of natural language is the high rate at which speakers produce novel expressions. Because of this novelty, a heavy-tail of rare events accounts for a significant amount of the total probability mass of distributions in language (Baayen, 2001). Standard language modeling metrics such as perplexity quantify the performance of language models (LM) in aggregate. As a result, we have relatively little understanding of whether neural LMs accurately estimate the probability of sequences in this heavy-tail of rare events. To address this gap, we develop a controlled evaluation scheme which uses generative models trained on natural data as artificial languages from which we can exactly compute sequence probabilities. Training LMs on generations from these artificial languages, we compare the sequence-level probability estimates given by LMs to the true probabilities in the target language. Our experiments reveal that LSTM and Transformer language models (i) systematically underestimate the probability of sequences drawn from the target language, and (ii) do so more severely for less-probable sequences. Investigating where this probability mass went, (iii) we find that LMs tend to overestimate the probability of ill formed (perturbed) sequences. In addition, we find that this underestimation behaviour (iv) is weakened, but not eliminated by greater amounts of training data, and (v) is exacerbated for target distributions with lower entropy.

Motivation & Objective

• To investigate how neural language models (LSTMs and Transformers) estimate the probability of rare, novel sequences in natural language, which are collectively significant despite individual low probability.
• To address the gap in understanding instance-level performance of LMs on the heavy-tail of language distributions, as standard metrics like perplexity fail to capture such behavior.
• To develop a controlled evaluation scheme using artificial languages generated from a pre-trained Transformer, enabling exact computation of true sequence probabilities.
• To analyze how training data size and target distribution properties (e.g., entropy) modulate the distortion in LM probability estimates.
• To determine where the under-estimated probability mass from rare, well-formed sequences is redirected, particularly in comparison to ill-formed (perturbed) sequences.

Proposed method

• Use a pre-trained Transformer language model (e.g., GPT-2) fine-tuned on natural language data as a generative model to define artificial languages with known, exact sequence probabilities.
• Sample sequences from this generative model to form training corpora for downstream LMs (LSTM and Transformer), creating controlled target distributions.
• Train separate LMs on these artificial corpora and compare their predicted sequence probabilities to the true probabilities from the generative model.
• Control the entropy of the generative model’s conditional distributions to create artificial languages with varying distributional properties (e.g., high vs. low entropy).
• Perform perturbation experiments by introducing small, syntactically ill-formed changes to well-formed sequences and compare LM probability estimates to the original true probabilities.
• Quantify model performance using sequence-level probability estimation error, with a focus on rare sequences and their deviation from the true distribution.

Experimental results

Research questions

• RQ1 Do neural language models systematically underestimate the probability of rare, well-formed sequences in natural language distributions?
• RQ2 How does the magnitude of underestimation vary with sequence rarity (i.e., inverse frequency)?
• RQ3 Where does the probability mass from under-estimated rare sequences go—does the model overestimate ill-formed or perturbed sequences?
• RQ4 How does increasing training data size affect the degree of probability distortion in neural LMs?
• RQ5 How do properties of the target distribution, such as entropy, influence the extent of underestimation in LM probability estimates?

Key findings

• LSTM and Transformer language models systematically underestimate the probability of well-formed sequences drawn from the target language, particularly for rarer sequences.
• The underestimation is more severe for less probable sequences, indicating a failure to adequately represent the tail of the distribution.
• The under-estimated probability mass is not redirected to high-frequency (head) sequences, but instead is over-allocated to ill-formed or perturbed sequences.
• Increasing training data reduces the magnitude of underestimation but does not eliminate it, suggesting the issue is inherent to the model architecture or training process.
• Underestimation is exacerbated in target distributions with lower entropy, indicating that models struggle more with structured, less random distributions.
• The results suggest that neural LMs spread probability mass too uniformly across sequence space, failing to concentrate it on high-probability, well-formed sequences in the tail of the distribution.