[Paper Review] Pre-training via Paraphrasing

Mike Lewis, Marjan Ghazvininejad+4 more|arXiv (Cornell University)|Jun 26, 2020

Topic Modeling37 references89 citations

TL;DR

MARGE is a multilingual retrieval-based pre-training model that learns to reconstruct targets by retrieving and conditioning on related documents, enabling strong zero-shot and fine-tuned performance across languages.

ABSTRACT

We introduce MARGE, a pre-trained sequence-to-sequence model learned with an unsupervised multi-lingual multi-document paraphrasing objective. MARGE provides an alternative to the dominant masked language modeling paradigm, where we self-supervise the reconstruction of target text by retrieving a set of related texts (in many languages) and conditioning on them to maximize the likelihood of generating the original. We show it is possible to jointly learn to do retrieval and reconstruction, given only a random initialization. The objective noisily captures aspects of paraphrase, translation, multi-document summarization, and information retrieval, allowing for strong zero-shot performance on several tasks. For example, with no additional task-specific training we achieve BLEU scores of up to 35.8 for document translation. We further show that fine-tuning gives strong performance on a range of discriminative and generative tasks in many languages, making MARGE the most generally applicable pre-training method to date.

Motivation & Objective

• Motivate a pre-training objective beyond masked language modeling by leveraging multilingual paraphrasing of related documents.
• Introduce MARGE, a retrieval-augmented sequence-to-sequence model trained to reconstruct target text from retrieved evidence documents.
• Demonstrate zero-shot and fine-tuned performance across translation, summarization, paraphrasing, and QA in many languages.
• Show that the model learns retrieval and reconstruction jointly from random initialization and without task-specific data for pre-training.

Proposed method

• Define a multi-source seq2seq model that encodes retrieved documents and decodes the target while conditioning on retrieved evidence.
• Learn a relevance scorer f(x, z) via a document encoder g with cosine similarity to bias cross-attention toward relevant retrieved documents.
• Train with an autoencoder-style reconstruction loss that conditions on retrieved documents and their relevance scores.
• Construct batches by retrieving and linking related documents within shards to maximize target-evidence connectivity.
• Implement a trainable bias in cross-attention that incorporates the retrieved document relevance f(xi, zj) during decoding.
• Pre-train at scale on CC-NEWS and Wikipedia with a large Transformer-based architecture and progressive optimization.

Experimental results

Research questions

• RQ1 Can a retrieval-based reconstruction objective serve as a viable pre-training alternative to masked language models?
• RQ2 To what extent can a joint retrieval and reconstruction model achieve zero-shot and fine-tuned performance across multilingual translation, summarization, and QA?
• RQ3 How does incorporating cross-language retrieved evidence affect cross-lingual transfer and document-level generation?
• RQ4 What are the limitations and domain dependencies of retrieval-based pre-training for multilingual settings?
• RQ5 How does the learned retrieval component behave across languages with varying data abundance and linguistic similarity?

Key findings

• MARGE achieves up to BLEU scores of 35.8 on document translation without task-specific fine-tuning.
• Zero-shot cross-lingual performance on summarization and translation tasks is competitive with strong baselines.
• Cross-lingual sentence retrieval on BUCC2018 and Tatoeba shows MARGE outperforming other unsupervised models.
• Paraphrase task (PAWS-X) shows MARGE achieving state-of-the-art-like results in zero-shot transfer.
• Question answering (MLQA) shows competitive results with XLM-R, with strong performance in Chinese.
• Fine-tuning MARGE yields competitive results against masked language models across multiple languages and tasks.