[Paper Review] Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs

Kuan-Hao Huang, Kai-Wei Chang|arXiv (Cornell University)|Jan 26, 2021

Topic Modeling59 references4 citations

TL;DR

This paper proposes SynPG, a neural paraphrase generator that disentangles semantics and syntax from unannotated text to enable syntactically controlled paraphrase generation without requiring annotated parallel pairs. By training on sentence reconstruction using unordered word bags and target parse trees, SynPG achieves superior syntactic control over unsupervised baselines and competitive performance compared to supervised models when large unannotated data is available.

ABSTRACT

Paraphrase generation plays an essential role in natural language process (NLP), and it has many downstream applications. However, training supervised paraphrase models requires many annotated paraphrase pairs, which are usually costly to obtain. On the other hand, the paraphrases generated by existing unsupervised approaches are usually syntactically similar to the source sentences and are limited in diversity. In this paper, we demonstrate that it is possible to generate syntactically various paraphrases without the need for annotated paraphrase pairs. We propose Syntactically controlled Paraphrase Generator (SynPG), an encoder-decoder based model that learns to disentangle the semantics and the syntax of a sentence from a collection of unannotated texts. The disentanglement enables SynPG to control the syntax of output paraphrases by manipulating the embedding in the syntactic space. Extensive experiments using automatic metrics and human evaluation show that SynPG performs better syntactic control than unsupervised baselines, while the quality of the generated paraphrases is competitive. We also demonstrate that the performance of SynPG is competitive or even better than supervised models when the unannotated data is large. Finally, we show that the syntactically controlled paraphrases generated by SynPG can be utilized for data augmentation to improve the robustness of NLP models.

Motivation & Objective

• To develop a paraphrase generation model that controls syntactic structure without relying on expensive annotated paraphrase pairs.
• To disentangle semantic meaning from syntactic structure in sentences using only unannotated text.
• To enable high-quality paraphrase generation with diverse syntactic forms through explicit syntactic control signals.
• To evaluate whether such a model can match or exceed supervised baselines in performance when trained on large unannotated corpora.
• To demonstrate the utility of SynPG-generated paraphrases for data augmentation to improve model robustness against syntactic adversarial attacks.

Proposed method

• Train a Transformer-based encoder-decoder model (SynPG) to reconstruct source sentences using two inputs: a bag-of-words representation (semantic encoder) and a target constituency parse (syntactic encoder).
• Use a semantic encoder that processes words in a permutation-invariant manner to extract pure semantic embeddings.
• Use a syntactic encoder to embed the target parse tree into a contextualized syntactic representation.
• Combine the semantic and syntactic representations in the decoder to generate a paraphrase that preserves meaning but follows the target syntactic structure.
• Train the model end-to-end via reconstruction loss on unannotated text, eliminating the need for parallel paraphrase pairs.
• Apply the model at inference time with a source sentence and a desired target parse to generate syntactically controlled paraphrases.

Experimental results

Research questions

• RQ1 Can syntactically controlled paraphrase generation be achieved without using annotated paraphrase pairs?
• RQ2 Can disentangling semantics and syntax from unannotated text enable effective syntactic control in paraphrase generation?
• RQ3 How does SynPG's performance in syntactic control compare to unsupervised baselines?
• RQ4 Can SynPG achieve competitive paraphrase quality compared to supervised models when large unannotated data is available?
• RQ5 Can SynPG-generated paraphrases improve the robustness of downstream NLP models through data augmentation?

Key findings

• SynPG outperforms unsupervised baselines in syntactic control, generating paraphrases that are significantly more aligned with the target parse structure.
• Human evaluation confirms that SynPG-generated paraphrases are more accurate in following syntactic specifications while maintaining competitive fluency and meaning preservation.
• With sufficient unannotated data, SynPG's performance matches or exceeds that of supervised models on paraphrase generation tasks.
• SynPG-generated paraphrases improve model robustness: on the GLUE benchmark, models trained with SynPG data augmentation showed significantly lower success rates in syntactic adversarial attacks (e.g., 33.9% brok vs. 58.3% for base model on RTE).
• The model effectively generates diverse paraphrases by manipulating syntactic representations while preserving semantic content, even without parallel supervision.
• The approach is particularly valuable in low-resource domains where annotated paraphrase pairs are scarce but unannotated text is abundant.