[Paper Review] ParaSCI: A Large Scientific Paraphrase Dataset for Longer Paraphrase Generation

Qingxiu Dong, Xiaojun Wan+1 more|arXiv (Cornell University)|Jan 21, 2021

Topic Modeling23 references4 citations

TL;DR

ParaSCI is a large-scale scientific paraphrase dataset comprising 350,044 paraphrase pairs from ACL and arXiv, constructed using intra- and inter-paper patterns such as repeated contributions and term definitions. It enables effective long-form scientific paraphrase generation, with models trained on ParaSCI producing longer, more diverse, and knowledge-rich outputs compared to those trained on general-domain datasets like Quora or MSCOCO.

ABSTRACT

We propose ParaSCI, the first large-scale paraphrase dataset in the scientific field, including 33,981 paraphrase pairs from ACL (ParaSCI-ACL) and 316,063 pairs from arXiv (ParaSCI-arXiv). Digging into characteristics and common patterns of scientific papers, we construct this dataset though intra-paper and inter-paper methods, such as collecting citations to the same paper or aggregating definitions by scientific terms. To take advantage of sentences paraphrased partially, we put up PDBERT as a general paraphrase discovering method. The major advantages of paraphrases in ParaSCI lie in the prominent length and textual diversity, which is complementary to existing paraphrase datasets. ParaSCI obtains satisfactory results on human evaluation and downstream tasks, especially long paraphrase generation.

Motivation & Objective

• To address the lack of large-scale, long-form paraphrase datasets in the scientific domain, which hinders scientific paraphrase generation.
• To develop a method for discovering paraphrases not only in full sentences but also in partially paraphrased segments, maximizing utilization of existing scientific text.
• To construct a high-quality, diverse, and long-form paraphrase dataset tailored for scientific NLP tasks, especially generation and data augmentation.
• To enable training of neural models that generate scientifically accurate, contextually rich, and structurally complex paraphrases.

Proposed method

• Construct ParaSCI by extracting paraphrase pairs using intra-paper methods (e.g., repeated contributions) and inter-paper methods (e.g., citations to the same work, definitions of the same term across papers).
• Design PDBERT, a fine-tuned BERT model that identifies semantically equivalent parts between two sentences, even when only partially paraphrased.
• Generate pseudo-training data by stitching existing paraphrase pairs to train PDBERT for partial paraphrase discovery.
• Filter and validate candidate paraphrase pairs using human evaluation and automatic metrics to ensure high quality.
• Release two subsets: ParaSCI-ACL (33,981 pairs) and ParaSCI-arXiv (316,063 pairs), both publicly available.
• Train and evaluate paraphrase generation models on ParaSCI and compare performance with models trained on Quora and MSCOCO.

Experimental results

Research questions

• RQ1 Can a large-scale, long-form paraphrase dataset be systematically constructed from scientific literature using intra- and inter-paper patterns?
• RQ2 Can a model like PDBERT effectively identify partial paraphrase segments in scientific text, improving data utilization?
• RQ3 Does training a paraphrase generation model on ParaSCI lead to better performance on long, complex scientific sentences compared to general-domain datasets?
• RQ4 To what extent do models trained on ParaSCI incorporate scientific terminology, abbreviations, and domain-specific knowledge in generated paraphrases?

Key findings

• ParaSCI contains 350,044 high-quality paraphrase pairs—33,981 from ACL and 316,063 from arXiv—making it the first large-scale scientific paraphrase dataset.
• Models trained on ParaSCI generate longer, more structurally complex sentences with richer scientific terminology and proper abbreviation usage, unlike models trained on Quora or MSCOCO, which produce short, incomplete, or structurally similar outputs.
• PDBERT successfully identifies semantically equivalent parts in partially paraphrased sentences, enabling the discovery of valuable paraphrase patterns that would otherwise be discarded.
• Human evaluation confirms that paraphrases from ParaSCI are more diverse and contextually appropriate, especially for long-form scientific content.
• Generated sentences from ParaSCI-trained models include domain-specific knowledge such as entity sizes (e.g., Penn Discourse Treebank having 2,312 articles) and technical abbreviations (e.g., 'moses' for phrase-based MT systems).
• The dataset demonstrates strong transferability, with models trained on ParaSCI outperforming those on general datasets in both automatic and human evaluations for long-form paraphrase generation.