[Paper Review] Universal Neural Machine Translation for Extremely Low Resource Languages

Jiatao Gu, Hany Hassan+2 more|arXiv (Cornell University)|Feb 15, 2018

Natural Language Processing Techniques24 references47 citations

TL;DR

The paper introduces Universal Lexical Representation (ULR) and Mixture of Language Experts (MoLE) to enable transfer learning across multiple languages, achieving strong BLEU gains for extremely low-resource language pairs using tiny parallel corpora and zero-shot transfer.

ABSTRACT

In this paper, we propose a new universal machine translation approach focusing on languages with a limited amount of parallel data. Our proposed approach utilizes a transfer-learning approach to share lexical and sentence level representations across multiple source languages into one target language. The lexical part is shared through a Universal Lexical Representation to support multilingual word-level sharing. The sentence-level sharing is represented by a model of experts from all source languages that share the source encoders with all other languages. This enables the low-resource language to utilize the lexical and sentence representations of the higher resource languages. Our approach is able to achieve 23 BLEU on Romanian-English WMT2016 using a tiny parallel corpus of 6k sentences, compared to the 18 BLEU of strong baseline system which uses multilingual training and back-translation. Furthermore, we show that the proposed approach can achieve almost 20 BLEU on the same dataset through fine-tuning a pre-trained multi-lingual system in a zero-shot setting.

Motivation & Objective

• Motivate and address translation for languages with minimal parallel data.
• Develop cross-lingual lexical and sentence-level sharing to improve low-resource translation.
• Evaluate a universal NMT framework on near-zero-resource language pairs and compare to multilingual baselines.
• Demonstrate the benefits of back-translation and fine-tuning in ultra-low-resource settings.

Proposed method

• Introduce Universal Lexical Representation (ULR) to map words from any language into a shared universal token space using a trainable embedding matrix E^U and a projection that aligns language-specific embeddings with universal tokens.
• Define a discrete universal token space (mostly English) and learn a shared transformation A to compute q(u|x) for distributing a source word x over universal tokens.
• Use a learned interpolation between language-specific embeddings and universal-token mixtures for frequent words (alpha/beta scheme).
• Propose Mixture of Language Experts (MoLE) after the encoder to model sentence-level variation with one expert per auxiliary language and a gating network to select experts, trained with a language-gated objective.
• Train within a multilingual NMT framework, employing auxiliary high-resource languages to assist extremely low-resource pairs, and optionally apply back-translation (BT) and fine-tuning on tiny datasets.

Experimental results

Research questions

• RQ1 Can universal lexical representations enable effective sharing across languages with scarce parallel data?
• RQ2 Does a mixture of language experts improve sentence-level sharing and translation quality for low-resource languages?
• RQ3 How do ULR and MoLE interact with back-translation and fine-tuning to close the gap to high-resource data?
• RQ4 What is the impact of auxiliary languages and language family relatedness on zero-resource translation performance?
• RQ5 Can fine-tuning pre-trained multilingual models yield substantial gains for ultra-low-resource language pairs?

Key findings

• ULR with MoLE yields additive improvements over multilingual NMT baselines across Ro-En, Lv-En, and Ko-En where parallel data is extremely limited.
• On Ro-En with 6k parallel sentences, ULR plus MoLE and BT reaches BLEU gains of up to 7.98 over the baseline and up to 22.92 with BT, versus 28.34 for full-data NMT.
• ULR enables better handling of unknown tokens by projecting unseen words into the universal token space, mitigating vocabulary gaps in zero-resource settings.
• MoLE activations reflect language relatedness; Ro exploits Romance languages while Lv benefits from Russian, and MoLE tends to select appropriate experts per context.
• Fine-tuning a pre-trained multilingual model with ultra-low-resource data can achieve substantial BLEU gains, approaching ~20 BLEU after a few epochs for Ro-En.