[Paper Review] BERT & Family Eat Word Salad: Experiments with Text Understanding

Ashim Gupta, Giorgi Kvernadze+1 more|arXiv (Cornell University)|Jan 10, 2021

Topic Modeling43 references24 citations

TL;DR

This paper demonstrates that state-of-the-art BERT-family models fail to recognize incoherent, word-salad inputs—such as alphabetically sorted or permuted sentences—consistently producing high-confidence predictions despite the absence of linguistic meaning. The key contribution is showing that models trained to explicitly reject invalid inputs via regularization or an extra 'invalid' class can maintain high performance on valid data while robustly detecting nonsense, revealing a critical flaw in current NLP models' understanding capabilities.

ABSTRACT

In this paper, we study the response of large models from the BERT family to incoherent inputs that should confuse any model that claims to understand natural language. We define simple heuristics to construct such examples. Our experiments show that state-of-the-art models consistently fail to recognize them as ill-formed, and instead produce high confidence predictions on them. As a consequence of this phenomenon, models trained on sentences with randomly permuted word order perform close to state-of-the-art models. To alleviate these issues, we show that if models are explicitly trained to recognize invalid inputs, they can be robust to such attacks without a drop in performance.

Motivation & Objective

• To investigate whether state-of-the-art BERT-based models can distinguish between meaningful and incoherent (word salad) inputs.
• To expose the vulnerability of these models to destructive input transformations that erase linguistic structure and meaning.
• To evaluate mitigation strategies that enable models to recognize and reject invalid inputs while preserving performance on valid data.

Proposed method

• Define nine destructive transformations (e.g., alphabetical sorting, word permutation, repetition) that render input text meaningless while preserving input format.
• Apply these transformations to test examples from GLUE benchmark tasks (NLI, paraphrase detection, sentiment analysis) to generate invalid inputs.
• Train and fine-tune RoBERTa models on datasets augmented with invalid examples using three mitigation strategies: entropic regularization, probability thresholding, and adding an explicit 'invalid' class.
• Measure model performance using accuracy on original data and detection rate of invalid inputs, with hyperparameters tuned on a validation set.
• Use temperature scaling for probability calibration and grid search to optimize thresholding parameters.
• Evaluate transferability of mitigation by training on one transformation and testing on others.

Experimental results

Research questions

• RQ1 Can BERT-family models reliably detect incoherent inputs such as alphabetically sorted or permuted word sequences?
• RQ2 To what extent do state-of-the-art models maintain high-confidence predictions on invalid inputs that lack semantic or syntactic structure?
• RQ3 Can models be trained to reject invalid inputs without sacrificing performance on standard, well-formed inputs?
• RQ4 Do mitigation strategies like entropic regularization or introducing an 'invalid' class improve robustness to destructive transformations?
• RQ5 Is the ability to detect invalid inputs transferable across different types of destructive transformations?

Key findings

• On the MNLI dataset, RoBERTa models retained the same entailment prediction with 95% average confidence even when hypotheses were alphabetically sorted, a transformation that renders them meaningless.
• For 79% of MNLI examples, the model predicted the same label (entailment, contradiction, or neutral) on alphabetically sorted hypotheses as on the original, despite the loss of linguistic coherence.
• Models fine-tuned on randomly permuted word order performed nearly as well as the original models on the original test sets, indicating that syntactic structure is not essential for high performance.
• Entropic regularization reduced agreement scores on invalid inputs to an average of 35%, significantly increasing uncertainty, though it caused a >1% drop in accuracy on valid data in some cases.
• The strategy of adding an explicit 'invalid' class (B + Invalid) achieved the best balance, maintaining high accuracy on valid data while correctly identifying 97.10% of invalid inputs across all transformations.
• Mitigation via the 'invalid' class was transferable: models trained on one transformation type generalized well to others, suggesting broad applicability.