[Paper Review] Parallel Data Augmentation for Formality Style Transfer

Yi Zhang, Tao Ge+1 more|arXiv (Cornell University)|May 14, 2020

Natural Language Processing Techniques37 references4 citations

TL;DR

This paper proposes novel data augmentation methods—formality discrimination (F-Dis), multi-task transfer (M-Task), and back translation—for formality style transfer (FST), significantly improving model performance by leveraging synthetic parallel data. When used to pre-train models on the GYAFC benchmark, the augmented data achieves state-of-the-art results in both automatic and human evaluations.

ABSTRACT

The main barrier to progress in the task of Formality Style Transfer is the inadequacy of training data. In this paper, we study how to augment parallel data and propose novel and simple data augmentation methods for this task to obtain useful sentence pairs with easily accessible models and systems. Experiments demonstrate that our augmented parallel data largely helps improve formality style transfer when it is used to pre-train the model, leading to the state-of-the-art results in the GYAFC benchmark dataset.

Motivation & Objective

• Address the critical bottleneck of limited parallel training data in formality style transfer (FST), where existing datasets like GYAFC contain only ~100K sentence pairs.
• Overcome data scarcity by generating diverse, high-quality synthetic parallel sentence pairs using accessible models and systems.
• Improve model generalization and reduce overfitting in FST by incorporating formality transfer knowledge not present in the original training data.
• Demonstrate that pre-training with augmented data leads to state-of-the-art performance on the GYAFC benchmark, surpassing existing methods in both automatic and human evaluation metrics.

Proposed method

• Apply back translation by training a formal-to-informal seq2seq model on the original parallel data and using it to generate synthetic informal sentences from formal inputs, forming new (informal, formal) pairs.
• Implement formality discrimination (F-Dis) by translating informal sentences through a pivot language (e.g., French, German, Chinese) and using a trained CNN-based formality discriminator to filter pairs where the retranslated sentence shows significant formality improvement.
• Use multi-task transfer (M-Task) by leveraging existing grammatical error correction (GEC) datasets to pre-train or fine-tune the FST model, exploiting the overlap between ungrammaticality in informal text and the correction patterns in GEC.
• Select augmented data using a thresholded formality score difference: $(m{s}_i, m{s}_i') \in \mathcal{T}_{\text{aug}}$ if $P_+(\bm{s}_i') - P_+(\bm{s}_i) \geq \sigma$, with $\sigma = 0.6$.
• Pre-train the final FST model on the combination of original and augmented data, enabling better generalization and improved performance on downstream evaluation.
• Evaluate the model using automatic metrics (BLEU) and human evaluation on formality, fluency, and meaning preservation, with results compared against strong baselines like NMT-MTL and GPT-CAT.

Experimental results

Research questions

• RQ1 Can data augmentation techniques significantly improve the performance of formality style transfer models despite limited original parallel data?
• RQ2 How effective are formality discrimination and multi-task transfer in generating high-quality synthetic parallel data that capture formality transfer knowledge not present in the original data?
• RQ3 What impact do different pivot languages (e.g., French, German, Chinese) have on the quality and quantity of augmented data in the F-Dis method?
• RQ4 Does pre-training with augmented data lead to measurable improvements in both automatic and human evaluation metrics for FST?
• RQ5 How does the proposed method compare to existing state-of-the-art models in terms of formality transfer quality and fluency preservation?

Key findings

• The proposed data augmentation methods—F-Dis, M-Task, and back translation—generate large-scale, high-quality synthetic parallel data, significantly expanding the training signal for FST.
• Using the augmented data for pre-training leads to state-of-the-art performance on the GYAFC benchmark, outperforming both baseline models and existing SOTA methods in automatic evaluation.
• Human evaluation shows that the proposed model achieves a formality score of 1.45 (p < 0.05), fluency of 1.85 (p < 0.05), and meaning preservation of 1.92 (p < 0.05), all significantly better than the original data.
• Among pivot languages, German-based F-Dis yields the best performance (BLEU: 75.18 on F&R), balancing data quality and size, while Chinese-based F-Dis produces the largest dataset (680k pairs) but with higher noise due to segmentation ambiguity.
• The model achieves a BLEU score of 74.52 on the E&M set and 75.18 on the F&R set when using F-Dis with German as the pivot, outperforming the original data (69.44 and 74.19, respectively).
• The combination of data augmentation and pre-training enables the model to generalize better, reducing overfitting and improving robustness in style transfer without requiring architectural changes.