[Paper Review] G-DAUG: Generative Data Augmentation for Commonsense Reasoning

Yiben Yang, Chaitanya Malaviya+7 more|arXiv (Cornell University)|Apr 24, 2020

Topic Modeling14 citations

TL;DR

G-DAUG^C proposes a generative data augmentation method that uses pretrained language models to synthesize diverse, high-quality training examples for commonsense reasoning, significantly improving performance on low-resource benchmarks. It outperforms back-translation-based methods and sets a new state-of-the-art on WinoGrande, CODAH, and CommonsenseQA while enhancing robustness to distribution shift and adversarial examples.

ABSTRACT

Recent advances in commonsense reasoning depend on large-scale human-annotated training data to achieve peak performance. However, manual curation of training examples is expensive and has been shown to introduce annotation artifacts that neural models can readily exploit and overfit on. We investigate G-DAUG^C, a novel generative data augmentation method that aims to achieve more accurate and robust learning in the low-resource setting. Our approach generates synthetic examples using pretrained language models, and selects the most informative and diverse set of examples for data augmentation. In experiments with multiple commonsense reasoning benchmarks, G-DAUG^C consistently outperforms existing data augmentation methods based on back-translation, and establishes a new state-of-the-art on WinoGrande, CODAH, and CommonsenseQA. Further, in addition to improvements in in-distribution accuracy, G-DAUG^C-augmented training also enhances out-of-distribution generalization, showing greater robustness against adversarial or perturbed examples. Our analysis demonstrates that G-DAUG^C produces a diverse set of fluent training examples, and that its selection and training approaches are important for performance. Our findings encourage future research toward generative data augmentation to enhance both in-distribution learning and out-of-distribution generalization.

Motivation & Objective

• To address the high cost and annotation artifacts in human-annotated commonsense reasoning datasets.
• To improve model generalization in low-resource settings where labeled data is scarce.
• To develop a data augmentation method that generates diverse, fluent, and informative synthetic examples.
• To enhance both in-distribution performance and out-of-distribution robustness in commonsense reasoning models.
• To reduce model overfitting to annotation artifacts by introducing synthetic data with varied linguistic patterns.

Proposed method

• Generates synthetic training examples using pretrained language models to expand the training data distribution.
• Applies a selection mechanism to retain the most informative and diverse examples from the generated set.
• Employs a curriculum-like training strategy that prioritizes high-quality synthetic examples during fine-tuning.
• Leverages the fluency and semantic diversity of generated examples to improve model generalization.
• Uses a confidence-based filtering step to exclude low-quality or irrelevant synthetic examples.
• Combines generative augmentation with standard fine-tuning to enhance performance on downstream benchmarks.

Experimental results

Research questions

• RQ1 Can generative data augmentation improve performance on commonsense reasoning benchmarks with limited labeled data?
• RQ2 Does G-DAUG^C produce more diverse and fluent training examples compared to back-translation-based methods?
• RQ3 To what extent does G-DAUG^C improve out-of-distribution generalization and robustness to adversarial examples?
• RQ4 How do the selection and training strategies in G-DAUG^C contribute to its performance gains?
• RQ5 Can synthetic data generated via large language models reduce overfitting to annotation artifacts in commonsense datasets?

Key findings

• G-DAUG^C achieves a new state-of-the-art on WinoGrande, outperforming previous methods with improved in-distribution accuracy.
• It sets new state-of-the-art results on CODAH and CommonsenseQA, demonstrating consistent gains across multiple benchmarks.
• The model fine-tuned with G-DAUG^C exhibits stronger robustness against adversarial and perturbed inputs compared to baselines.
• The method produces a diverse set of fluent synthetic examples, which contributes to better generalization.
• The selection and training procedures in G-DAUG^C are critical for performance, as ablation studies show significant drops without them.
• G-DAUG^C reduces overfitting to annotation artifacts by introducing linguistic variation through generative augmentation.