[Paper Review] Linguistically-Informed Transformations (LIT): A Method for Automatically Generating Contrast Sets

Chuanrong Li, Lin Shengshuo+4 more|arXiv (Cornell University)|Oct 16, 2020

Topic Modeling22 references4 citations

TL;DR

This paper proposes Linguistically-Informed Transformations (LIT), a method that uses a linguistic grammar (ERG) to automatically generate high-quality contrast sets for NLP models by applying targeted syntactic and semantic perturbations. The approach enables fine-grained probing of model robustness, reveals systematic failures in BERT and RoBERTa on simple linguistic phenomena, and improves model generalization through data augmentation without harming in-distribution performance.

ABSTRACT

Although large-scale pretrained language models, such as BERT and RoBERTa, have achieved superhuman performance on in-distribution test sets, their performance suffers on out-of-distribution test sets (e.g., on contrast sets). Building contrast sets often re-quires human-expert annotation, which is expensive and hard to create on a large scale. In this work, we propose a Linguistically-Informed Transformation (LIT) method to automatically generate contrast sets, which enables practitioners to explore linguistic phenomena of interests as well as compose different phenomena. Experimenting with our method on SNLI and MNLI shows that current pretrained language models, although being claimed to contain sufficient linguistic knowledge, struggle on our automatically generated contrast sets. Furthermore, we improve models' performance on the contrast sets by apply-ing LIT to augment the training data, without affecting performance on the original data.

Motivation & Objective

• To address the lack of large-scale, high-quality contrast sets for evaluating NLP model robustness beyond in-distribution performance.
• To overcome the high cost and limited scalability of human-annotated contrast sets in NLP.
• To enable fine-grained analysis of model failures on specific linguistic phenomena such as tense mismatches, passivization, and modality.
• To improve model generalization by augmenting training data with linguistically informed transformations.
• To demonstrate that traditional linguistic knowledge remains essential for evaluating and enhancing data-driven NLP models.

Proposed method

• LIT uses the Meaning-Text Theory-based ERG grammar to parse input sentences and identify linguistic phenomena amenable to transformation.
• It applies a set of predefined linguistic transformations—such as tense shifting, passivization, and it-clefting—based on syntactic and semantic rules from the grammar.
• Transformations are applied systematically to generate contrastive examples that preserve syntactic structure while altering meaning or label, ensuring minimal perturbation.
• The method supports compositional generalization by combining multiple transformations on a single example, enabling complex probing scenarios.
• LIT generates multiple transformation variants per input, allowing for diverse and controlled evaluation of model behavior.
• The approach is applied to SNLI and MNLI datasets, with human evaluation confirming high quality of generated contrast sets.

Experimental results

Research questions

• RQ1 Can linguistically informed, automatic transformation methods generate high-quality contrast sets that reveal model weaknesses not exposed by standard benchmarks?
• RQ2 How do state-of-the-art NLP models like BERT and RoBERTa perform on contrast sets generated via LIT, particularly on simple and compositional linguistic perturbations?
• RQ3 To what extent can data augmentation using LIT improve model robustness on out-of-distribution test sets without degrading in-distribution performance?
• RQ4 Which linguistic phenomena are underrepresented in existing NLI datasets like SNLI and MNLI, and how does this affect model generalization?
• RQ5 Does fine-tuning on LIT-augmented data lead to systematic understanding of linguistic transformations, as evidenced by performance on compositional generalization tasks?

Key findings

• BERT and RoBERTa show significant performance drops on LIT-generated contrast sets, even on simple transformations like tense mismatches, indicating poor robustness to basic linguistic perturbations.
• Models fine-tuned on LIT-augmented data improve performance on out-of-distribution test sets without sacrificing accuracy on original in-distribution data.
• RoBERTa achieves strong zero-shot performance on compositional transformations (e.g., combining passivization and future tense) when fine-tuned on simple transformations, suggesting systematic learning of transformation patterns.
• Human evaluation confirms that LIT-generated contrast sets are high-quality, with strong semantic and syntactic fidelity and correct label changes.
• Analysis of SNLI and MNLI reveals significant data bias: only 2.95% of MNLI sentences are in future tense, and passive-future or future-passive pairs are absent, explaining model failures on these transformations.
• The method successfully transforms approximately 20% of SNLI instances, with coverage limited by grammar parsing success, but the results demonstrate proof-of-concept for integrating linguistic knowledge into data augmentation.