[Paper Review] UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

Hangbo Bao, Li Dong+9 more|arXiv (Cornell University)|Feb 28, 2020

Topic Modeling36 references225 citations

TL;DR

UniLMv2 pre-trains a unified bidirectional encoder and a sequence-to-sequence decoder using pseudo-masked language modeling, achieving state-of-the-art results on multiple NLU and NLG benchmarks.

ABSTRACT

We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM). Given an input text with masked tokens, we rely on conventional masks to learn inter-relations between corrupted tokens and context via autoencoding, and pseudo masks to learn intra-relations between masked spans via partially autoregressive modeling. With well-designed position embeddings and self-attention masks, the context encodings are reused to avoid redundant computation. Moreover, conventional masks used for autoencoding provide global masking information, so that all the position embeddings are accessible in partially autoregressive language modeling. In addition, the two tasks pre-train a unified language model as a bidirectional encoder and a sequence-to-sequence decoder, respectively. Our experiments show that the unified language models pre-trained using PMLM achieve new state-of-the-art results on a wide range of natural language understanding and generation tasks across several widely used benchmarks.

Motivation & Objective

• Motivate unified pre-training to support both language understanding and generation in a single model.
• Introduce pseudo-masked language modeling (PMLM) to learn inter- and intra-relations among tokens and spans.
• Reuse encoding results to avoid redundant computation and enable efficient joint AE and PAR learning.
• Show that combining autoencoding and partially autoregressive objectives yields complementary representations for diverse tasks.

Proposed method

• Define two MLM objectives: autoencoding (AE) for bidirectional encoding and partially autoregressive (PAR) for sequence-to-sequence decoding.
• Introduce pseudo masks [Pseudo] that share position embeddings with original tokens to enable common context across AE and PAR in a single forward pass.
• Use blockwise masking and factorization orders to allow spans as prediction units in PAR (not just single tokens).
• Design self-attention masks to control context accessibility and prevent information leakage across factorization steps.
• Train with L = L_AE + L_PAR to jointly optimize both objectives and reuse hidden states across tasks.
• Fine-tune on NLU tasks as a bidirectional encoder and on NLG tasks as an autoregressive decoder with task-specific outputs.

Experimental results

Research questions

• RQ1 Can a unified pre-training framework that combines AE and PAR via PMLM improve both understanding and generation tasks?
• RQ2 How do masking strategies and factorization orders affect learning of inter- and intra-token/span relations?
• RQ3 Does sharing parameters between AE and PAR yield efficiency gains without sacrificing performance?
• RQ4 What are the empirical gains on SQuAD, GLUE, and abstractive summarization benchmarks when using UniLMv2?

Key findings

• On SQuAD, UniLMv2 base achieves 93.1 F1 and 87.1 EM on v1.1, and 86.1 F1 and 83.3 EM on v2.0.
• On GLUE, UniLMv2 base outperforms BERT-base and XLNet-base across tasks like MNLI and SST-2, with notable gains in MNLI accuracy (88.5) and SST-2 accuracy (95.1).
• For abstractive summarization, UniLMv2 base reaches strong ROUGE scores on CNN/DailyMail and XSum, outperforming several baselines and comparably sized models.
• Question generation and other NLG tasks show UniLMv2 base delivering superior BLEU/METEOR/ROUGE metrics relative to baselines and some larger pretrained models.
• Ablation studies indicate that AE+PAR joint training yields the best results among pre-training objectives, with PAR contributing strong span-level modeling and block masking enhancing long-distance dependency capture.