[Paper Review] Unsupervised Parallel Corpus Mining on Web Data

Guokun Lai, Zihang Dai+1 more|arXiv (Cornell University)|Sep 18, 2020

Natural Language Processing Techniques12 references4 citations

TL;DR

This paper proposes an unsupervised web parallel corpus mining pipeline that eliminates the need for labeled parallel data by leveraging an unsupervised machine translator to generate pseudo-parallel data for training a classifier that filters high-quality parallel sentences from web crawls. The method achieves state-of-the-art results on low-resource language pairs (39.81 BLEU on En-Ro, 38.95 on Ro-En), matching or surpassing supervised systems on WMT benchmarks.

ABSTRACT

With a large amount of parallel data, neural machine translation systems are able to deliver human-level performance for sentence-level translation. However, it is costly to label a large amount of parallel data by humans. In contrast, there is a large-scale of parallel corpus created by humans on the Internet. The major difficulty to utilize them is how to filter them out from the noise website environments. Current parallel data mining methods all require labeled parallel data as the training source. In this paper, we present a pipeline to mine the parallel corpus from the Internet in an unsupervised manner. On the widely used WMT'14 English-French and WMT'16 English-German benchmarks, the machine translator trained with the data extracted by our pipeline achieves very close performance to the supervised results. On the WMT'16 English-Romanian and Romanian-English benchmarks, our system produces new state-of-the-art results, 39.81 and 38.95 BLEU scores, even compared with supervised approaches.

Motivation & Objective

• To address the high cost and scarcity of human-annotated parallel corpora for training neural machine translation systems.
• To enable large-scale, low-cost parallel corpus mining from the web without requiring any labeled parallel data.
• To improve the performance of neural machine translation models on low-resource language pairs using unsupervised mining.
• To develop a pipeline that is scalable and applicable to any language pair lacking parallel data.

Proposed method

• Train an unsupervised machine translation model using XLM-style objectives, including denoising autoencoding and back-translation.
• Generate a seed dictionary from the pseudo-parallel data produced by the unsupervised translator to guide web crawling.
• Use the Bitextor crawler with document and sentence alignment to extract raw parallel sentence pairs from multilingual websites.
• Apply a classifier trained on the pseudo-parallel data to filter out low-quality and noisy sentence pairs from the crawled corpus.
• Use the final filtered corpus as supervised data to fine-tune a neural machine translation model.
• Integrate the pipeline with unsupervised training objectives to evaluate performance in low-resource settings.

Experimental results

Research questions

• RQ1 Can an unsupervised machine translation system be used to bootstrap the mining of high-quality parallel corpora from the web without any labeled data?
• RQ2 How effective is the proposed pipeline in extracting usable parallel data compared to supervised methods?
• RQ3 Can the mined parallel data achieve performance comparable to or better than fully supervised systems on low-resource language pairs?
• RQ4 What is the impact of the filtering step on model performance, and how robust is training to noisy data?

Key findings

• The proposed pipeline achieves 39.81 BLEU on the WMT’16 English-Romanian translation benchmark, setting a new state-of-the-art result.
• For Romanian-English translation, the system achieves 38.95 BLEU, surpassing the best supervised result (38.5 BLEU) on the same benchmark.
• On the WMT’14 English-French and WMT’16 English-German benchmarks, the model trained on mined data achieves performance very close to that of fully supervised systems.
• The ablation study shows that while raw crawled data performs nearly as well as filtered data, low-quality data significantly degrades performance, confirming the classifier’s effectiveness.
• The method reduces dependency on labeled data and enables high-quality parallel corpus mining for any language pair, even in low-resource settings.