[Paper Review] Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation

Toan Nguyen, David Chiang|arXiv (Cornell University)|Aug 31, 2017

Natural Language Processing TechniquesComputer Science8 references112 citations

TL;DR

The paper extends Zoph et al.'s transfer learning by sharing source vocabularies through Byte Pair Encoding (BPE) and transliteration to leverage related low-resource Turkic languages, improving NMT performance especially when using BPE.

ABSTRACT

We present a simple method to improve neural translation of a low-resource language pair using parallel data from a related, also low-resource, language pair. The method is based on the transfer method of Zoph et al., but whereas their method ignores any source vocabulary overlap, ours exploits it. First, we split words using Byte Pair Encoding (BPE) to increase vocabulary overlap. Then, we train a model on the first language pair and transfer its parameters, including its source word embeddings, to another model and continue training on the second language pair. Our experiments show that transfer learning helps word-based translation only slightly, but when used on top of a much stronger BPE baseline, it yields larger improvements of up to 4.3 BLEU.

Motivation & Objective

• Motivate improvement of NMT for very low-resource language pairs using related languages.
• Explore whether transferring from a related low-resource parent language helps a low-resource child language.
• Investigate the impact of vocabulary overlap and subword segmentation on transfer learning effectiveness.
• Assess whether combining transfer learning with BPE yields better gains than either approach alone.

Proposed method

• Use a two-layer attentional NMT model with Luong-style attention.
• Follow Zoph et al. transfer: initialize child from a trained parent model, then fine-tune on child data; transfer source embeddings but freeze target embeddings.
• Increase cross-language vocabulary overlap by transliterating to a common script (Uyghur transliterated to Latin) and applying BPE on the union of parent and child vocabularies.
• Learn BPE merge rules from the union of source and target data of both languages to produce a single shared vocabulary for both models.
• Train word-based baselines and BPE-based systems, exploring different vocabulary sizes and BPE operation counts (5k–60k).
• Evaluate with tokenized BLEU on dev/test, applying length normalization during decoding.

Experimental results

Research questions

• RQ1 Does transfer learning from a related, low-resource parent language improve translation quality for a low-resource child language?
• RQ2 Does sharing a BPE-based subword vocabulary between parent and child improve transfer effectiveness compared to naive transfer or no transfer?
• RQ3 How does transliteration to a common script affect vocabulary overlap and transfer gains?
• RQ4 Is the boost from transfer learning additive when combined with a strong BPE baseline?

Key findings

• Word-based transfer provides limited and inconsistent gains in low-resource settings.
• BPE-based systems outperform word-based systems, and transfer on top of a strong BPE baseline yields larger improvements.
• Uzbek as a parent language to Turkish and Uyghur yields improvements over the BPE baseline of up to 0.8 BLEU (Turkish-English) and 4.3 BLEU (Uyghur-English).
• Transfer with shared vocabulary (via BPE) produces consistent improvements over both the baseline and non-shared transfer setups.
• The combination of BPE and transfer is more effective than either approach alone, likely due to increased lexical overlap from subword units.