[Paper Review] Unified Language Model Pre-training for Natural Language Understanding and Generation

Li Dong|arXiv (Cornell University)|May 8, 2019

Topic Modeling48 references949 citations

TL;DR

UniLM unifies pre-training for NLU and NLG by using a single Transformer with configurable self-attention masks to support bidirectional, unidirectional, and sequence-to-sequence objectives, achieving strong results across GLUE, SQuAD 2.0, CoQA, and multiple generation tasks.

ABSTRACT

This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks. The model is pre-trained using three types of language modeling tasks: unidirectional, bidirectional, and sequence-to-sequence prediction. The unified modeling is achieved by employing a shared Transformer network and utilizing specific self-attention masks to control what context the prediction conditions on. UniLM compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks. Moreover, UniLM achieves new state-of-the-art results on five natural language generation datasets, including improving the CNN/DailyMail abstractive summarization ROUGE-L to 40.51 (2.04 absolute improvement), the Gigaword abstractive summarization ROUGE-L to 35.75 (0.86 absolute improvement), the CoQA generative question answering F1 score to 82.5 (37.1 absolute improvement), the SQuAD question generation BLEU-4 to 22.12 (3.75 absolute improvement), and the DSTC7 document-grounded dialog response generation NIST-4 to 2.67 (human performance is 2.65). The code and pre-trained models are available at https://github.com/microsoft/unilm.

Motivation & Objective

• Motivate a single, unified pre-training framework to cover both NLU and NLG tasks.
• Reduce the complexity and resource needs of maintaining separate LMs for different tasks.
• Learn generalizable representations via joint optimization over multiple language modeling objectives.
• Demonstrate strong performance on benchmarks spanning understanding (GLUE, QA) and generation (summarization, Q&A, dialog).

Proposed method

• Use a multi-layer Transformer with shared parameters across all LM objectives.
• Design and apply three types of cloze-based pre-training tasks: unidirectional LM, bidirectional LM, and sequence-to-sequence LM.
• Implement self-attention masks to control context accessibility for each token during prediction.
• Pre-train with a balanced mix of objectives (1/3 bidirectional, 1/3 sequence-to-sequence, 1/3 unidirectional in each batch).
• Fine-tune the pre-trained model for downstream tasks by adapting the masking scheme and adding task-specific layers as needed.
• Initialize from BERT_LARGE and train on English Wikipedia and BookCorpus; use WordPiece tokenization and a 28,996-vocabulary.
• Pre-training details: 24-layer Transformer, 1024 hidden size, 16 heads, about 340M parameters; training on ~770k steps with 8 V100 GPUs; learning rate 3e-5, warmup 40k steps, dropout 0.1.

Experimental results

Research questions

• RQ1 Can a single pre-trained Transformer be effectively fine-tuned for both NLU and NLG tasks using unified objectives?
• RQ2 Do unified pre-training objectives lead to competitive or superior performance compared with task-specific LMs on standard benchmarks?
• RQ3 To what extent can a model pre-trained with multiple LM objectives improve performance on generation tasks such as abstractive summarization, question generation, and dialog response generation?
• RQ4 What are the comparative gains on structured QA benchmarks (SQuAD 2.0, CoQA) and generation benchmarks when using UniLM?

Key findings

• UniLM compares favorably with BERT on GLUE and outperform on SQuAD 2.0 and CoQA extractive QA.
• On five NLG datasets, UniLM achieves new state-of-the-art results: CNN/DailyMail abstractive summarization ROUGE-L of 40.51, Gigaword ROUGE-L of 35.75, CoQA generative QA F1 of 82.5, SQuAD QG BLEU-4 of 22.12, and DSTC7 dialog response generation NIST-4 of 2.67 (human 2.65).
• In abstractive summarization, UniLM surpasses prior abstractive systems and even a strong extractive model on CNN/DailyMail (ROUGE-L improvement of 0.88 over the best extractive).
• In generative QA on CoQA, UniLM achieves substantial gains over Seq2Seq and PGNet baselines (F1 82.5).
• For question generation on SQuAD, UniLM achieves BLEU-4 22.12, METEOR 25.06, ROUGE-L 51.07, outperforming prior models.
• GLUE results show UniLM achieving comparable performance to BERT_LARGE on several tasks (Table 11).
• The model can serve as an encoder for NLU and as an encoder-decoder for NLG, enabling cross-task transfer via unified pre-training.