[Paper Review] ParaCotta: Synthetic Multilingual Paraphrase Corpora from the Most Diverse Translation Sample Pair

Alham Fikri Aji, Tirana Noor Fatyanosa+7 more|arXiv (Cornell University)|May 10, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes ParaCotta, a method to generate synthetic multilingual paraphrase corpora using only monolingual text and a neural machine translation (NMT) system. By generating multiple translations via beam search and selecting the pair with the lowest BLEU score (indicating highest lexical diversity), the approach produces semantically similar yet lexically diverse paraphrase pairs across 17 languages, releasing a publicly available dataset that outperforms baseline methods in human evaluation and semantic similarity while maintaining strong lexical diversity.

ABSTRACT

We release our synthetic parallel paraphrase corpus across 17 languages: Arabic, Catalan, Czech, German, English, Spanish, Estonian, French, Hindi, Indonesian, Italian, Dutch, Romanian, Russian, Swedish, Vietnamese, and Chinese. Our method relies only on monolingual data and a neural machine translation system to generate paraphrases, hence simple to apply. We generate multiple translation samples using beam search and choose the most lexically diverse pair according to their sentence BLEU. We compare our generated corpus with the exttt{ParaBank2}. According to our evaluation, our synthetic paraphrase pairs are semantically similar and lexically diverse.

Motivation & Objective

• To address the scarcity of high-quality multilingual paraphrase datasets beyond English by creating a scalable, low-resource method.
• To generate paraphrase pairs that are both semantically similar and lexically diverse without requiring parallel bilingual corpora.
• To develop a method that relies solely on monolingual data and a pre-trained NMT system, enabling broad language coverage.
• To release a publicly available synthetic paraphrase corpus across 17 languages for downstream NLP tasks.
• To evaluate the trade-off between lexical diversity and semantic similarity in synthetic paraphrase generation.

Proposed method

• Generate multiple translation samples from a monolingual English sentence using beam search with a pre-trained English-to-target-language NMT model.
• Select the pair of translations with the lowest BLEU score as the final paraphrase pair, under the assumption that lower BLEU indicates higher lexical diversity.
• Use a monolingual English corpus (e.g., ParaBank2, Wikipedia, NewsCrawl, Tatoeba) as input for paraphrase generation.
• Apply the same method to generate paraphrases for 17 languages: Arabic, Catalan, Czech, German, English, Spanish, Estonian, French, Hindi, Indonesian, Italian, Dutch, Romanian, Russian, Swedish, Vietnamese, and Chinese.
• Filter generated pairs using BLEU thresholds (e.g., 20–60 or 20–80) to balance semantic similarity and lexical diversity.
• Train a Transformer-based seq2seq model on the synthetic paraphrase corpus to evaluate its performance against baseline methods like round-trip machine translation.

Experimental results

Research questions

• RQ1 Can a synthetic paraphrase corpus be effectively generated using only monolingual data and a pre-trained NMT system, without requiring parallel bilingual corpora?
• RQ2 Does selecting the most lexically diverse translation pair (via low BLEU) yield paraphrase pairs that are both semantically similar and lexically diverse?
• RQ3 How does the quality of the synthetic paraphrase corpus compare to existing benchmarks like ParaBank2 in terms of semantic similarity and lexical diversity?
• RQ4 To what extent can BLEU-based filtering improve the balance between semantic similarity and lexical diversity in synthetic paraphrase pairs?
• RQ5 Can the proposed method generate high-quality paraphrase data across a wide range of low-resource and high-resource languages?

Key findings

• The ParaCotta-generated paraphrase corpus achieves a human-annotated semantic similarity score of 88.9 on the Indonesian dataset when filtered with BLEU 20–80, outperforming both unfiltered ParaCotta and round-trip MT.
• The filtered ParaCotta corpus maintains high semantic similarity (sBERT cosine similarity of 92.1) while achieving a BLEU score of 48.0, indicating strong lexical diversity.
• The method produces paraphrase pairs that are significantly more diverse than those in ParaBank2, as measured by BLEU and Jaccard index, while still maintaining high semantic similarity.
• A correlation was observed between lower BLEU scores and higher lexical diversity, confirming the effectiveness of using BLEU as a proxy for diversity in the selection process.
• The approach outperforms round-trip machine translation in both semantic similarity and lexical diversity, with the latter achieving a semantic similarity score of only 78.1.
• The released dataset spans 17 languages and is publicly available at https://github.com/afaji/paracotta-paraphrase, enabling broader use in multilingual NLP tasks.