[Paper Review] Adversarial Training for Large Neural Language Models

Xiaodong Liu, Hao Cheng+5 more|arXiv (Cornell University)|Apr 20, 2020

Adversarial Robustness in Machine Learning59 references91 citations

TL;DR

The paper introduces ALUM, a general adversarial training algorithm for pre-training and fine-tuning large neural language models, showing improved generalization and robustness across NLP tasks, including with RoBERTa.

ABSTRACT

Generalization and robustness are both key desiderata for designing machine learning methods. Adversarial training can enhance robustness, but past work often finds it hurts generalization. In natural language processing (NLP), pre-training large neural language models such as BERT have demonstrated impressive gain in generalization for a variety of tasks, with further improvement from adversarial fine-tuning. However, these models are still vulnerable to adversarial attacks. In this paper, we show that adversarial pre-training can improve both generalization and robustness. We propose a general algorithm ALUM (Adversarial training for large neural LangUage Models), which regularizes the training objective by applying perturbations in the embedding space that maximizes the adversarial loss. We present the first comprehensive study of adversarial training in all stages, including pre-training from scratch, continual pre-training on a well-trained model, and task-specific fine-tuning. ALUM obtains substantial gains over BERT on a wide range of NLP tasks, in both regular and adversarial scenarios. Even for models that have been well trained on extremely large text corpora, such as RoBERTa, ALUM can still produce significant gains from continual pre-training, whereas conventional non-adversarial methods can not. ALUM can be further combined with task-specific fine-tuning to attain additional gains. The ALUM code is publicly available at https://github.com/namisan/mt-dnn.

Motivation & Objective

• Motivate the need for models that generalize well and resist adversarial attacks in NLP.
• Propose a unified adversarial training framework (ALUM) applicable to pre-training and fine-tuning.
• Evaluate ALUM across a broad set of NLP tasks and datasets.
• Demonstrate that adversarial pre-training yields gains over strong baselines (e.g., BERT, RoBERTa) and can be combined with adversarial fine-tuning.

Proposed method

• Adopt perturbations in embedding space to maximize adversarial loss (virtual adversarial training).
• Regularize the training objective with an adversarial term controlling label smoothness in embedding neighborhoods (alpha parameter).
• Use a curriculum approach: first standard training, then continue with adversarial training (ALUM).
• Estimate perturbations with a small inner loop (K=1) for efficiency.
• Apply ALUM to pre-training from scratch, continual pre-training, and task-specific fine-tuning, on transformer-based models such as BERT and RoBERTa.
• Provide results showing substantial gains over non-adversarial baselines and compatibility with task-specific fine-tuning.

Experimental results

Research questions

• RQ1 Does adversarial pre-training improve generalization on standard NLP benchmarks?
• RQ2 Does adversarial pre-training also improve robustness to adversarial or challenging evaluation datasets?
• RQ3 Can ALUM yield gains when continually pre-training already well-trained models (e.g., RoBERTa)?
• RQ4 What is the effect of combining adversarial pre-training with adversarial fine-tuning across tasks?

Key findings

• Adversarial pre-training with ALUM improves both generalization and robustness across a wide range of NLP tasks.
• ALUM yields gains over BERT on SQuAD and MNLI, and outperforms standard baselines even with models trained on large corpora like RoBERTa.
• Continual pre-training with ALUM provides gains for RoBERTa where standard continual pre-training does not.
• ALUM improves robustness on adversarial benchmarks such as ANLI, Adversarial SQuAD, and HELLASWAG, often outperforming strong baselines.
• Combining adversarial pre-training with adversarial fine-tuning yields the best results on several tasks (e.g., MNLI, ANLI, SciTail, SWAG, HELLASWAG).
• The approach is applicable to pre-training from scratch, continual pre-training, and task-specific fine-tuning, and is demonstrated on BERT and RoBERTa.