[Paper Review] Data Augmentation for Neural Machine Translation using Generative Language Model

Seok Jin Oh, Su ah Lee+1 more|arXiv (Cornell University)|Jul 26, 2023

Natural Language Processing TechniquesComputer Science3 citations

TL;DR

This paper proposes prompt-based data augmentation for Neural Machine Translation using GPT-3.5-turbo to generate synthetic parallel sentences without additional training costs. By comparing three prompt strategies—paraphrasing, multi-target translation, and storytelling—it demonstrates that storytelling yields the most diverse data, improving BLEU score by 0.68 over the baseline, with the best performance at 29.17 BLEU when data is doubled.

ABSTRACT

Despite the rapid growth in model architecture, the scarcity of large parallel corpora remains the main bottleneck in Neural Machine Translation. Data augmentation is a technique that enhances the performance of data-hungry models by generating synthetic data instead of collecting new ones. We explore prompt-based data augmentation approaches that leverage large-scale language models such as ChatGPT. To create a synthetic parallel corpus, we compare 3 methods using different prompts. We employ two assessment metrics to measure the diversity of the generated synthetic data. This approach requires no further model training cost, which is mandatory in other augmentation methods like back-translation. The proposed method improves the unaugmented baseline by 0.68 BLEU score.

Motivation & Objective

• To address the scarcity of large-scale parallel corpora in Neural Machine Translation (NMT), especially for low-resource languages or domains.
• To explore prompt-based data augmentation using large generative language models like ChatGPT to generate synthetic parallel data without fine-tuning.
• To evaluate the impact of different prompt designs—paraphrasing, multi-target translation, and storytelling—on data diversity and model performance.
• To measure the diversity of generated synthetic data using cosine similarity and BLEU scores between original and synthetic sentences.
• To demonstrate that data diversity is crucial for improving NMT performance, especially when domain shifts or data scarcity occur.

Proposed method

• Three prompt-based methods are used to generate synthetic parallel data: paraphrasing the source and target sentences, generating multiple translations for each source sentence (multi-target), and generating a three-sentence story in the source language followed by translation (storytelling).
• The GPT-3.5-turbo model is prompted via the OpenAI API using specific templates tailored to each method, with prompts designed to preserve meaning while increasing variation.
• Sentence embeddings are computed using the LASER encoder to calculate cosine similarity between original and synthetic sentence pairs as a diversity metric.
• Lexical similarity is measured via BLEU scores between original and synthetic target sentences to assess lexical divergence.
• The mBART-50 model is fine-tuned on original and augmented data, with evaluation using SacreBLEU on a held-out test set.
• Experiments vary the augmentation ratio from 0.5x to 3.0x the original training data size, and the best checkpoint is selected based on validation BLEU scores.

Experimental results

Research questions

• RQ1 Does prompt-based data augmentation using a generative language model improve NMT performance without additional training costs?
• RQ2 How do different prompt strategies—paraphrasing, multi-target translation, and storytelling—affect the diversity and quality of synthetic parallel data?
• RQ3 To what extent does data diversity, measured via embedding similarity and BLEU scores, correlate with improved NMT model performance?
• RQ4 Can storytelling-based augmentation mitigate out-of-domain hallucinations and improve robustness in low-resource settings?
• RQ5 Which prompt design yields the highest BLEU score improvement over the unaugmented baseline?

Key findings

• The storytelling method achieved the highest BLEU score of 29.17 when synthetic data was augmented at 2.0x the original size, outperforming the baseline of 28.49.
• The paraphrase and multi-target methods showed decreasing BLEU scores with increased augmentation, indicating limited data diversity and potential overfitting.
• The storytelling method produced the lowest cosine similarity (0.596) and BLEU score (2.908) between original and synthetic sentences, confirming higher data diversity.
• The paraphrase method had the highest cosine similarity (0.900) and BLEU score (23.409), indicating minimal variation from original sentences.
• The multi-target method showed moderate diversity with cosine similarity of 0.825 and BLEU of 15.543, but still underperformed storytelling in model performance.
• The study confirms that generating diverse synthetic data is essential for narrowing the gap between training data distribution and real-world language distribution in NMT.