[Paper Review] Cross-lingual Retrieval for Iterative Self-Supervised Training

Chau Tran, Yuqing Tang+2 more|arXiv (Cornell University)|Jun 16, 2020

Topic Modeling55 references48 citations

TL;DR

CRISS iteratively mines pseudo-parallel data using encoder outputs from multilingual models to improve cross-lingual alignment and translation without labeled parallel data, achieving SOTA in unsupervised MT and cross-lingual retrieval.

ABSTRACT

Recent studies have demonstrated the cross-lingual alignment ability of multilingual pretrained language models. In this work, we found that the cross-lingual alignment can be further improved by training seq2seq models on sentence pairs mined using their own encoder outputs. We utilized these findings to develop a new approach -- cross-lingual retrieval for iterative self-supervised training (CRISS), where mining and training processes are applied iteratively, improving cross-lingual alignment and translation ability at the same time. Using this method, we achieved state-of-the-art unsupervised machine translation results on 9 language directions with an average improvement of 2.4 BLEU, and on the Tatoeba sentence retrieval task in the XTREME benchmark on 16 languages with an average improvement of 21.5% in absolute accuracy. Furthermore, CRISS also brings an additional 1.8 BLEU improvement on average compared to mBART, when finetuned on supervised machine translation downstream tasks.

Motivation & Objective

• Demonstrate that encoder outputs from multilingual denoising autoencoders form language-agnostic representations that improve cross-lingual tasks.
• Show that finetuning on a single parallel language pair can boost alignment across many language directions.
• Develop an iterative mining-training loop that jointly discovers pseudo-parallel data and improves a multilingual model.
• Evaluate CRISS on unsupervised and supervised MT as well as sentence retrieval to establish performance gains.
• Provide insights and ablations on pretraining, multilingual versus bilingual training, and pivot-language choices.

Proposed method

• Use mBART as the initial multilingual Seq2Seq pretraining model to obtain language-agnostic sentence representations.
• Mine pseudo-parallel data by computing a margin-based score over KNN neighborhoods of sentence embeddings and selecting high-scoring pairs (Algorithm 1).
• Iteratively train a multilingual transformer on the mined data and repeat mining with the improved model (Algorithm 2).
• Augment mined pairs with a target-language token to enable multilingual MT finetuning.
• Limit mining directions to a subset (e.g., 90 language pairs) to achieve strong results with manageable computation.
• Evaluate using unsupervised MT benchmarks, Tatoeba XTREME-style sentence retrieval, and supervised MT finetuning.

Experimental results

Research questions

• RQ1 Can cross-lingual sentence representations learned by a multilingual denoising autoencoder be improved through self-mined parallel data?
• RQ2 Does finetuning on parallel data from a single language pair enhance cross-lingual alignment for all directions?
• RQ3 What is the impact of iterative mining-training cycles on unsupervised MT and cross-lingual retrieval performance?
• RQ4 How do multilingual versus bilingual fine-tuning strategies compare when using mined pseudo-parallel data?
• RQ5 What is the effect of the number of pivot languages on retrieval and translation quality?

Key findings

• CRISS achieves state-of-the-art unsupervised MT results across 9 language directions with an average BLEU gain of 2.4 over prior methods.
• On XTREME Tatoeba retrieval across 16 languages, CRISS yields an average absolute accuracy improvement of 21.5%.
• CRISS provides an average 1.8 BLEU improvement over mBART when finetuned on supervised MT downstream tasks.
• Finetuning on a single language pair can improve cross-lingual alignment for all directions; iterative mining further enhances performance.
• Multilingual training generally outperforms bilingual training for the mined pseudo-parallel data in downstream MT tasks.
• Using a small number of pivot languages (found optimal around 2, but tested up to 4) yields near-optimal gains with increased computational cost.