[Paper Review] Unsupervised Paraphrase Generation using Pre-trained Language Models

Chaitra V. Hegde, Shrikumar Patil|arXiv (Cornell University)|Jun 9, 2020

Topic ModelingComputer Science28 references46 citations

TL;DR

The paper presents an unsupervised approach to paraphrase generation by fine-tuning GPT-2 on a sentence reconstruction task using corrupted inputs, achieving high-quality and diverse paraphrases that improve downstream classification when used for data augmentation.

ABSTRACT

Large scale Pre-trained Language Models have proven to be very powerful approach in various Natural language tasks. OpenAI's GPT-2 \cite{radford2019language} is notable for its capability to generate fluent, well formulated, grammatically consistent text and for phrase completions. In this paper we leverage this generation capability of GPT-2 to generate paraphrases without any supervision from labelled data. We examine how the results compare with other supervised and unsupervised approaches and the effect of using paraphrases for data augmentation on downstream tasks such as classification. Our experiments show that paraphrases generated with our model are of good quality, are diverse and improves the downstream task performance when used for data augmentation.

Motivation & Objective

• Demonstrate that large pre-trained language models can generate quality paraphrases without labeled data.
• Show that reconstructing sentences from corrupted input can train paraphrase generation in an unsupervised way.
• Assess paraphrase quality, diversity, and usefulness for data augmentation on downstream tasks.

Proposed method

• Use GPT-2 fine-tuned on a sentence reconstruction task where the source is a corrupted sentence S (stop-words removed, 20% word shuffle, 20% synonym replacement) and the target is the original sentence T.
• Concatenate Source and Target with a [SEP] token to form input X for GPT-2 during training.
• Generate multiple paraphrases per input using top-k sampling (k=10).
• Filter paraphrases by semantic similarity to the input using Sentence Transformers and a cosine similarity threshold of 0.75.
• Evaluate quality with METEOR and ROUGE-L, diversity with self-BLEU, and usefulness via data augmentation on SST-2 and TREC tasks.

Experimental results

Research questions

• RQ1 Can unsupervised GPT-2-based paraphrasing yield paraphrases of high quality without labeled paraphrase data?
• RQ2 Do paraphrases produced via this method improve downstream classification performance when used for data augmentation?
• RQ3 Are the generated paraphrases diverse yet semantically faithful to the source?
• RQ4 How does the unsupervised approach compare to supervised and other unsupervised paraphrasing methods?

Key findings

• Paraphrases are of high quality, with METEOR outperforming several supervised models in the reported results.
• The method yields diverse paraphrases, indicated by low self-BLEU scores among candidates.
• Paraphrases provide measurable improvements in downstream SST-2 and TREC classification tasks when used for data augmentation.
• Human evaluation shows paraphrase correctness at 75.5% accuracy on average.
• Compared to unsupervised baselines, the proposed approach achieves competitive ROUGE-1 and ROUGE-2 scores and superior METEOR scores.