[Paper Review] Detecting and Exorcising Statistical Demons from Language Models with Anti-Models of Negative Data

Michael Wick, Kate Silverstein+3 more|arXiv (Cornell University)|Oct 22, 2020

Topic Modeling31 references4 citations

TL;DR

This paper proposes a method to detect and mitigate harmful statistical biases—'statistical demons'—in neural language models by training anti-models on negative data distributions derived from positive training data. By using unsmoothed n-gram models to generate negative training examples, the approach reduces reliance on spurious n-gram patterns and improves syntactic generalization, achieving up to 46% error reduction on challenging subject-verb agreement tasks in LSTMs and GPT-2.

ABSTRACT

It's been said that "Language Models are Unsupervised Multitask Learners." Indeed, self-supervised language models trained on "positive" examples of English text generalize in desirable ways to many natural language tasks. But if such models can stray so far from an initial self-supervision objective, a wayward model might generalize in undesirable ways too, say to nonsensical "negative" examples of unnatural language. A key question in this work is: do language models trained on (positive) training data also generalize to (negative) test data? We use this question as a contrivance to assess the extent to which language models learn undesirable properties of text, such as n-grams, that might interfere with the learning of more desirable properties of text, such as syntax. We find that within a model family, as the number of parameters, training epochs, and data set size increase, so does a model's ability to generalize to negative n-gram data, indicating standard self-supervision generalizes too far. We propose a form of inductive bias that attenuates such undesirable signals with negative data distributions automatically learned from positive data. We apply the method to remove n-gram signals from LSTMs and find that doing so causes them to favor syntactic signals, as demonstrated by large error reductions (up to 46% on the hardest cases) on a syntactic subject-verb agreement task.

Motivation & Objective

• To investigate whether language models trained on positive data also generalize to undesirable 'negative' data, such as ungrammatical or statistically biased sequences.
• To identify and quantify the extent to which self-supervised language models learn spurious n-gram patterns instead of syntactic structure.
• To develop a method that detects and attenuates these undesirable statistical signals using automatically learned negative data distributions.
• To evaluate whether removing such signals improves the model's ability to learn linguistically meaningful representations, particularly syntax.

Proposed method

• Train an unsmoothed tri-gram language model on the same positive training data to serve as a detector of common n-gram patterns.
• Use this tri-gram model to generate 'negative' data—ungrammatical or statistically biased sequences—that reflect the undesirable patterns the main model might overfit to.
• Introduce a regularization term during fine-tuning that penalizes the main model (e.g., LSTM or GPT-2) for assigning high probability to these negative sequences.
• Apply a hyperparameter α to control the strength of attenuation, allowing for a trade-off between reducing n-gram bias and preserving performance on grammatical data.
• Evaluate the method on syntactic probing tasks, particularly subject-verb agreement, using standard benchmarks like Marvin & Linzen’s data.
• Measure performance via negative log-likelihood (NLL) on both positive and negative dev sets to assess generalization and bias mitigation.

Experimental results

Research questions

• RQ1 To what extent do self-supervised language models generalize to ungrammatical or statistically biased negative data?
• RQ2 How does increasing model size, training data, and training epochs affect a model’s tendency to learn and rely on spurious n-gram patterns?
• RQ3 Can we automatically learn negative data distributions from positive data to serve as a signal for detecting and mitigating undesirable statistical biases?
• RQ4 Does attenuating n-gram signals through negative data improve the model’s ability to learn syntactic generalizations?
• RQ5 What is the trade-off between improved long-distance syntactic generalization and potential degradation on short-distance dependencies?

Key findings

• Fine-tuning with tri-gram attenuation significantly improved performance on long-distance syntactic dependencies, with up to a 46% relative error reduction on the hardest subject-verb agreement cases.
• The method reduced reliance on n-gram patterns: negative dev perplexity increased with higher α, indicating the model was less likely to assign high probability to ungrammatical sequences.
• On the 'across a subject relative clause' construction, the model achieved a 95% accuracy (NLL of 0.95) with α=1, outperforming the baseline GPT-2 score of 0.72.
• For reflexive anaphora and negative polarity items, the method improved performance on complex, long-distance cases, with NLL scores rising from 0.76 to 0.79 on 'across a relative clause (RA)' and from 0.83 to 0.94 on 'across a relative clause (NPI)' with α=8.
• Performance on short-distance dependencies such as 'in an object relative clause' decreased with increasing α, indicating a trade-off between long- and short-distance generalization.
• The method demonstrated that models can be guided toward syntactic generalization by actively suppressing statistical noise, even when such noise is highly predictive in the training data.