[Paper Review] AR: Auto-Repair the Synthetic Data for Neural Machine Translation

Shanbo Cheng, Shaohui Kuang+4 more|arXiv (Cornell University)|Apr 5, 2020

Natural Language Processing Techniques25 references4 citations

TL;DR

This paper proposes Auto-Repair (AR), a seq2seq-based framework that improves synthetic parallel data for neural machine translation by learning to correct noisy translations generated via back-translation or forward-translation. By training separate source-to-source and target-to-target repair models on monolingual data, AR significantly enhances translation quality, achieving up to 2.07 BLEU gains on WMT14 EN→DE and 3.85 BLEU on IWSLT14 DE→EN over baseline models.

ABSTRACT

Compared with only using limited authentic parallel data as training corpus, many studies have proved that incorporating synthetic parallel data, which generated by back translation (BT) or forward translation (FT, or selftraining), into the NMT training process can significantly improve translation quality. However, as a well-known shortcoming, synthetic parallel data is noisy because they are generated by an imperfect NMT system. As a result, the improvements in translation quality bring by the synthetic parallel data are greatly diminished. In this paper, we propose a novel Auto- Repair (AR) framework to improve the quality of synthetic data. Our proposed AR model can learn the transformation from low quality (noisy) input sentence to high quality sentence based on large scale monolingual data with BT and FT techniques. The noise in synthetic parallel data will be sufficiently eliminated by the proposed AR model and then the repaired synthetic parallel data can help the NMT models to achieve larger improvements. Experimental results show that our approach can effective improve the quality of synthetic parallel data and the NMT model with the repaired synthetic data achieves consistent improvements on both WMT14 EN!DE and IWSLT14 DE!EN translation tasks.

Motivation & Objective

• To address the persistent issue of noisy synthetic parallel data in neural machine translation, which limits performance gains from data augmentation.
• To develop a low-cost, scalable method to improve synthetic data quality without iterative retraining or complex data filtering.
• To enable direct repair of synthetic sentences using monolingual data and pre-trained NMT models, enhancing downstream translation quality.

Proposed method

• Train separate S2S and T2T auto-repair models to transform low-quality synthetic sentences into higher-quality ones using monolingual data.
• Use the pre-trained S2T and T2S models to generate synthetic data from monolingual sources, then apply the AR models to refine them.
• Integrate both original and repaired synthetic data into the final NMT training process, combining authentic, FT, BT, and repaired FT/BT data.
• Apply a warm-up learning rate schedule with Adam optimization and beam search decoding during training.
• Evaluate repair quality using BLEU, change rate, and better rate metrics on dev sets to measure improvement over raw synthetic data.
• Use a dual-stage training strategy: first train AR models on synthetic-to-authentic sentence pairs, then fine-tune the main NMT model with combined data.

Experimental results

Research questions

• RQ1 Can a seq2seq-based auto-repair model effectively improve the quality of synthetic parallel data generated via back-translation or forward-translation?
• RQ2 Does repairing synthetic data lead to consistent improvements in NMT performance across different language pairs and datasets?
• RQ3 How much of the synthetic data is actually modified by the AR model, and what proportion of repairs result in higher BLEU scores compared to original synthetic sentences?
• RQ4 Can the AR framework be applied efficiently to various types of synthetic data without requiring expensive iterative retraining?
• RQ5 Does combining original synthetic data with repaired synthetic data yield better NMT performance than using only original synthetic data?

Key findings

• The BASE + BTR-ADD + FTR-ADD model achieved a 2.07 BLEU point improvement over the baseline on the WMT14 EN→DE translation task.
• On the IWSLT14 DE→EN task, the best AR model outperformed the baseline by 3.85 BLEU points and the strong synthetic baseline (BT+FT) by 1.19 BLEU points.
• The AR model improved the BLEU score of synthetic data by over 11 points on both EN→DE and DE→EN tasks, demonstrating effective quality enhancement.
• More than 76% of synthetic sentences were modified by the AR model, indicating substantial learning of transformation patterns.
• Over 69% of repaired sentences achieved a higher sentence-level BLEU score than their original synthetic counterparts, confirming consistent quality gains.
• The AR framework achieved an average improvement of 2.96 BLEU points over the baseline and 0.97 BLEU points over the strong synthetic data baseline across both tasks.