[Paper Review] BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Michael Lewis, Yinhan Liu+6 more|arXiv (Cornell University)|Oct 29, 2019

Topic ModelingComputer Science235 citations

TL;DR

BART is a denoising autoencoder pretraining framework that combines bidirectional encoding with autoregressive decoding, enabling strong performance across generation and comprehension tasks, and competitive results on discriminative tasks.

ABSTRACT

We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and many other more recent pretraining schemes. We evaluate a number of noising approaches, finding the best performance by both randomly shuffling the order of the original sentences and using a novel in-filling scheme, where spans of text are replaced with a single mask token. BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks. It matches the performance of RoBERTa with comparable training resources on GLUE and SQuAD, achieves new state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains of up to 6 ROUGE. BART also provides a 1.1 BLEU increase over a back-translation system for machine translation, with only target language pretraining. We also report ablation experiments that replicate other pretraining schemes within the BART framework, to better measure which factors most influence end-task performance.

Motivation & Objective

• Motivate a versatile pretraining objective that supports both generation and comprehension tasks.
• Explore a wide range of text corruption (noising) schemes and identify which yield robust downstream performance.
• Show how a single pretrained model can be fine-tuned for diverse tasks (classification, QA, generation, translation).
• Demonstrate that denoising seq2seq pretraining can match or exceed existing strong pretraining methods on multiple benchmarks.

Proposed method

• Use a standard Transformer-based seq2seq architecture with a bidirectional encoder and autoregressive decoder.
• Pretrain by corrupting documents with arbitrary noising functions and training to reconstruct the original text (negative log-likelihood).
• Evaluate multiple noising schemes including token masking, token deletion, text infilling, sentence permutation, document rotation, and their combinations.
• Fine-tune for sequence classification, token classification, sequence generation, and machine translation with task-specific adaptations.
• For translation, extend BART with a small additional encoder to map foreign words to English, training end-to-end with a separate vocabulary.

Experimental results

Research questions

• RQ1 Can a denoising autoencoder pretraining objective that operates on corrupted text generalize across generation and comprehension tasks?
• RQ2 Which noising schemes yield robust end-task performance across a diverse set of NLP benchmarks?
• RQ3 How does BART compare to existing pretraining approaches (e.g., BERT, RoBERTa, XLNet) on both discriminative and generative tasks?
• RQ4 Can a single pretrained model improve machine translation when used as a decoder with an added encoder?
• RQ5 What ablations reveal about the contribution of different pretraining objectives to downstream performance?

Key findings

• BART matches RoBERTa-like performance on GLUE and SQuAD for discriminative tasks while delivering state-of-the-art results on abstractive tasks.
• Text infilling and related noising schemes yield consistently strong performance across tasks, outperforming other pretraining objectives in many settings.
• On summarization, BART substantially outperforms prior work, with notable gains on abstractive datasets like XSum.
• In translation, using BART as a pretrained decoder (with a small added encoder) yields BLEU gains over strong back-translation baselines.
• Ablation studies show that preprocessing choices and pretraining objectives influence end-task performance, with bidirectional encoders and autoregressive decoders being advantageous for generation.