[Paper Review] TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP

John X. Morris, Eli Lifland+4 more|arXiv (Cornell University)|Apr 29, 2020

Adversarial Robustness in Machine LearningComputer Science35 references60 citations

TL;DR

TextAttack is a Python framework that unifies NLP adversarial attacks, data augmentation, and adversarial training by composing attacks from four modular components and enabling benchmarking across models and datasets.

ABSTRACT

While there has been substantial research using adversarial attacks to analyze NLP models, each attack is implemented in its own code repository. It remains challenging to develop NLP attacks and utilize them to improve model performance. This paper introduces TextAttack, a Python framework for adversarial attacks, data augmentation, and adversarial training in NLP. TextAttack builds attacks from four components: a goal function, a set of constraints, a transformation, and a search method. TextAttack's modular design enables researchers to easily construct attacks from combinations of novel and existing components. TextAttack provides implementations of 16 adversarial attacks from the literature and supports a variety of models and datasets, including BERT and other transformers, and all GLUE tasks. TextAttack also includes data augmentation and adversarial training modules for using components of adversarial attacks to improve model accuracy and robustness. TextAttack is democratizing NLP: anyone can try data augmentation and adversarial training on any model or dataset, with just a few lines of code. Code and tutorials are available at https://github.com/QData/TextAttack.

Motivation & Objective

• Motivate robust evaluation of NLP models by standardizing adversarial attacks.
• Lower barriers to reproducing and comparing attack methods across datasets and models.
• Provide a unified platform to reuse attack components for new methods.
• Enable data augmentation and adversarial training to improve accuracy and robustness.

Proposed method

• Attack construction via four modular components: goal function, constraints, transformation, and search method.
• Supports 16 implemented adversarial attacks from the literature as reusable attack recipes.
• Integrated with HuggingFace transformers and NLP datasets for easy testing across models and tasks.
• Includes data augmentation and adversarial training pipelines to improve model robustness.
• Presents a command-line and Python API for rapid experimentation (e.g., textattack attack, textattack train).

Experimental results

Research questions

• RQ1 How can NLP adversarial attacks be standardized and benchmarked across models and datasets?
• RQ2 Can a modular framework facilitate rapid construction of new attacks by reusing existing components?
• RQ3 What is the impact of data augmentation and adversarial training on model robustness across NLP tasks?
• RQ4 How broadly can attack recipes be applied to models beyond classification/entailment (e.g., translation, summarization, GLUE tasks)?

Key findings

• TextAttack provides implementations of 16 literature attacks as reusable recipes.
• It supports a wide range of models (including LSTM, CNN, BERT, RoBERTa) and NLP tasks (including all GLUE tasks).
• The framework enables data augmentation and adversarial training to improve robustness.
• Attack components are designed to be combined to reproduce or extend existing attacks with minimal code.
• The framework emphasizes usability, including integration with HuggingFace libraries and a claim of performing common tasks in under five lines of code.