[Paper Review] ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training

Weizhen Qi, Yu Yan+6 more|arXiv (Cornell University)|Jan 13, 2020

Topic Modeling45 references83 citations

TL;DR

ProphetNet introduces a future n-gram prediction objective with n-stream self-attention for Seq2Seq pre-training, achieving state-of-the-art results on abstractive summarization and question generation. It predicts multiple future tokens at each step and can be converted to a standard Transformer decoder for inference.

ABSTRACT

This paper presents a new sequence-to-sequence pre-training model called ProphetNet, which introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of optimizing one-step-ahead prediction in the traditional sequence-to-sequence model, the ProphetNet is optimized by n-step ahead prediction that predicts the next n tokens simultaneously based on previous context tokens at each time step. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent overfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large-scale dataset (160GB), respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for abstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new state-of-the-art results on all these datasets compared to the models using the same scale pre-training corpus.

Motivation & Objective

• Motivate improving Seq2Seq pre-training by planning for future tokens rather than only predicting the next token.
• Introduce future n-gram prediction as a self-supervised objective to reduce overfitting to local correlations.
• Develop an n-stream self-attention mechanism to enable simultaneous forecasting of multiple future tokens.
• Ensure the model remains compatible with standard inference by disabling predicting streams during generation.
• Demonstrate effectiveness through extensive ablations and comparisons on benchmark NLG tasks.

Proposed method

• Extend Transformer encoder-decoder with an extra set of predicting streams for future tokens (n-stream self-attention).
• Define future n-gram loss as a combination of the traditional language modeling loss and losses for predicting the next n-1 future tokens (with attenuation weights).
• Train with a denoising auto-encoder objective (mask-based span masking) adapted to predict n-grams within masked spans.
• Pre-train on 16GB (base) and 160GB (large) corpora using similar settings to MASS/BART/T5, with 512 input length and span masking.
• At inference, disable predicting streams so the model reduces to standard next-token prediction.
• Fine-tune on CNN/DailyMail, Gigaword, and SQuAD 1.1 QG tasks to assess generation quality.

Experimental results

Research questions

• RQ1 Does predicting future n-grams improve long-term dependency capture and global coherence in generated text compared to one-step-ahead prediction?
• RQ2 Can an n-stream self-attention decoder effectively learn to predict multiple future tokens during training while remaining compatible with standard inference?
• RQ3 How does ProphetNet perform on abstractive summarization and question generation benchmarks relative to other pre-trained Seq2Seq models?

Key findings

• ProphetNet achieves state-of-the-art ROUGE scores on CNN/DailyMail with 43.68 (R-1), 20.64 (R-2), and 40.72 (R-L).
• On Gigaword, ProphetNet outperforms baselines across metrics (highest reported in Table 2).
• For SQuAD 1.1 question generation, ProphetNet attains leading BLEU/METEOR/ROUGE scores compared to prior methods.
• Large-scale pre-training (160GB) yields further gains, achieving SOTA on CNN/DailyMail and Gigaword with substantially less pre-training data than some baselines.
• Without pre-training, ProphetNet still improves over Transformer baselines on CNN/DailyMail.
• Compared across n-gram settings, 2-gram and 3-gram variants outperform MASS and 1-gram baselines, with 2-gram offering favorable speed-accuracy trade-offs.