[Paper Review] Diversify Your Datasets: Analyzing Generalization via Controlled Variance in Adversarial Datasets

Ohad Rozen, Vered Shwartz+2 more|arXiv (Cornell University)|Oct 21, 2019

Machine Learning in Materials Science25 references4 citations

TL;DR

This paper proposes a controlled variance methodology to assess whether NLP models generalize over linguistic phenomena like dative alternation and numerical reasoning, rather than overfitting to specific adversarial datasets. By systematically varying syntactic complexity and lexical diversity (e.g., number ranges) in training and test sets, the authors show that BERT-based models fail to generalize across different syntactic structures or number ranges, revealing inherent model weaknesses beyond dataset blind spots.

ABSTRACT

Phenomenon-specific "adversarial" datasets have been recently designed to perform targeted stress-tests for particular inference types. Recent work (Liu et al., 2019a) proposed that such datasets can be utilized for training NLI and other types of models, often allowing to learn the phenomenon in focus and improve on the challenge dataset, indicating a "blind spot" in the original training data. Yet, although a model can improve in such a training process, it might still be vulnerable to other challenge datasets targeting the same phenomenon but drawn from a different distribution, such as having a different syntactic complexity level. In this work, we extend this method to drive conclusions about a model's ability to learn and generalize a target phenomenon rather than to "learn" a dataset, by controlling additional aspects in the adversarial datasets. We demonstrate our approach on two inference phenomena - dative alternation and numerical reasoning, elaborating, and in some cases contradicting, the results of Liu et al.. Our methodology enables building better challenge datasets for creating more robust models, and may yield better model understanding and subsequent overarching improvements.

Motivation & Objective

• To assess whether neural NLP models generalize over linguistic phenomena or merely overfit to specific adversarial datasets.
• To extend the inoculation methodology of Liu et al. (2019a) by controlling variance in linguistic dimensions such as syntactic complexity and lexical diversity.
• To determine whether model improvements on challenge datasets reflect true learning of a phenomenon or distribution-specific memorization.
• To guide the design of more robust, generalizable training data for specific inference phenomena.
• To identify inherent model weaknesses that persist even after fine-tuning on phenomenon-specific data.

Proposed method

• The authors generate synthetic adversarial datasets using a templating method that controls variance along linguistic dimensions like syntactic complexity and lexical diversity (e.g., number ranges).
• They fine-tune a BERT-based NLI model on training sets with controlled variance and evaluate performance on test sets drawn from different distributions.
• For dative alternation, they vary syntactic structures (e.g., double object vs. prepositional phrase constructions) while preserving semantic content.
• For numerical reasoning, they vary the range of numbers used in arithmetic comparisons (e.g., 30–49, 60–79, 200–299) while keeping syntax identical.
• Performance is measured as accuracy on test sets with varying distributions to assess generalization across syntactic and lexical dimensions.
• The approach enables isolating generalization failure due to distribution shift from failure due to model inductive bias.

Experimental results

Research questions

• RQ1 Can NLP models generalize over syntactic complexity when trained on adversarial datasets for dative alternation?
• RQ2 Does the model’s performance on numerical reasoning generalize across different number ranges, or is it limited to the training range?
• RQ3 To what extent does successful fine-tuning on a challenge dataset indicate true learning of a phenomenon versus memorization of distribution-specific patterns?
• RQ4 How do controlled variations in linguistic dimensions (syntax, lexical diversity) reveal model generalization limits?
• RQ5 Can the inoculation approach be misleading if the model fails to generalize across different data distributions?

Key findings

• The model shows significant difficulty generalizing over syntactic complexity in dative alternation, requiring extensive training on diverse syntactic structures to achieve stable performance.
• Performance drops by up to 3.2% when testing on more complex syntax than seen in training, indicating poor generalization across syntactic variation.
• For numerical reasoning, the model fails to generalize across different number ranges, achieving less than 70% accuracy on out-of-distribution ranges even after 20,000 training examples.
• The model’s performance on the same number range during training and testing is substantially higher, suggesting memorization of specific number pairs rather than learning of arithmetic rules.
• The results contradict the original inoculation analysis, indicating that success on a single challenge dataset may not reflect true generalization but rather dataset-specific overfitting.
• The study reveals an inherent model weakness in handling numerical reasoning across diverse number ranges, highlighting a critical limitation for robust NLI systems.