[Paper Review] Extract and Edit: An Alternative to Back-Translation for Unsupervised Neural Machine Translation

Jiawei Wu, Xin Wang+1 more|arXiv (Cornell University)|Apr 4, 2019

Natural Language Processing Techniques38 references4 citations

TL;DR

This paper proposes Extract-Edit, a novel unsupervised neural machine translation method that replaces back-translation by extracting and editing real target-language sentences to serve as high-quality training signals. By introducing a comparative translation loss that optimizes translations relative to these real sentence pairs, the approach achieves state-of-the-art performance across four language pairs, improving BLEU scores by 2.0 to 3.63 points over prior methods.

ABSTRACT

The overreliance on large parallel corpora significantly limits the applicability of machine translation systems to the majority of language pairs. Back-translation has been dominantly used in previous approaches for unsupervised neural machine translation, where pseudo sentence pairs are generated to train the models with a reconstruction loss. However, the pseudo sentences are usually of low quality as translation errors accumulate during training. To avoid this fundamental issue, we propose an alternative but more effective approach, extract-edit, to extract and then edit real sentences from the target monolingual corpora. Furthermore, we introduce a comparative translation loss to evaluate the translated target sentences and thus train the unsupervised translation systems. Experiments show that the proposed approach consistently outperforms the previous state-of-the-art unsupervised machine translation systems across two benchmarks (English-French and English-German) and two low-resource language pairs (English-Romanian and English-Russian) by more than 2 (up to 3.63) BLEU points.

Motivation & Objective

• To address the fundamental limitation of back-translation in unsupervised NMT, where accumulated translation errors degrade model performance due to low-quality pseudo-parallel sentences.
• To develop a method that leverages real, high-quality sentence pairs from monolingual target corpora to guide unsupervised training, avoiding distributional drift.
• To introduce a comparative translation loss that evaluates translations based on relative similarity to extracted-and-edited real sentences, improving alignment and generalization.
• To enable fully unsupervised training without requiring parallel parallel data, making the approach applicable to low-resource language pairs.

Proposed method

• Extract potential parallel sentence candidates from the target monolingual corpus using semantic similarity between source sentences and target sentences.
• Apply an editing mechanism to revise the extracted sentences so they align semantically with the source sentence, improving fluency and relevance.
• Formulate a comparative translation loss that maximizes the similarity of the model’s translation to the best among the extracted-and-edited target sentences, relative to other candidates.
• Jointly train the extract-edit module and the NMT model in an end-to-end, fully unsupervised manner using only monolingual monolingual corpora.
• Use a dual-attention mechanism to align source and target sentences during extraction, ensuring semantic consistency.
• Leverage language modeling and initialization techniques (e.g., bilingual dictionary projection) to stabilize training in the absence of parallel data.

Experimental results

Research questions

• RQ1 Can real sentence pairs extracted from monolingual target corpora serve as more reliable training signals than back-translated pseudo-pairs in unsupervised NMT?
• RQ2 Does a comparative translation loss that evaluates translations relative to multiple extracted-and-edited candidates improve model performance compared to standard reconstruction loss?
• RQ3 How does the extract-edit framework perform across diverse language pairs, including low-resource ones like English-Romanian and English-Russian?
• RQ4 To what extent does the method mitigate distributional shift caused by error accumulation in back-translation?
• RQ5 Can the extract-edit approach generalize to unsupervised learning beyond machine translation?

Key findings

• The extract-edit method consistently outperforms the previous state-of-the-art unsupervised NMT system (Lample et al., 2018b) across all evaluated language pairs.
• On the English-French and English-German benchmarks, the method achieves improvements of over 2.0 BLEU points, with gains up to 3.63 BLEU points.
• The approach demonstrates strong generalization on low-resource language pairs, including English-Romanian and English-Russian, where performance gains are significant despite limited parallel data.
• The comparative translation loss effectively guides the model to generate translations that are more aligned with real, high-quality target sentences than with pseudo-pairs.
• The method reduces distributional drift by relying on real target sentences, avoiding the error accumulation inherent in back-translation.
• Empirical results confirm that extract-edit is more effective than back-translation in both high- and low-resource settings, validating its robustness and scalability.