[Paper Review] A Teacher-Student Framework for Zero-Resource Neural Machine Translation

Yun Chen, Yang Liu+2 more|arXiv (Cornell University)|May 2, 2017

Natural Language Processing Techniques29 references104 citations

TL;DR

The paper introduces a teacher-student framework to train a source-to-target NMT model without parallel data by using a pivot-to-target teacher model to guide learning on a source-pivot corpus, improving zero-resource translation quality over pivot-based baselines.

ABSTRACT

While end-to-end neural machine translation (NMT) has made remarkable progress recently, it still suffers from the data scarcity problem for low-resource language pairs and domains. In this paper, we propose a method for zero-resource NMT by assuming that parallel sentences have close probabilities of generating a sentence in a third language. Based on this assumption, our method is able to train a source-to-target NMT model ("student") without parallel corpora available, guided by an existing pivot-to-target NMT model ("teacher") on a source-pivot parallel corpus. Experimental results show that the proposed method significantly improves over a baseline pivot-based model by +3.0 BLEU points across various language pairs.

Motivation & Objective

• Motivate zero-resource NMT due to limited parallel data for low-resource language pairs.
• Propose a teacher-student framework that uses a pivot-to-target teacher to guide a source-to-target student model.
• Validate assumptions that parallel sentences have similar probabilities to generate a third-language sentence, enabling knowledge transfer.
• Demonstrate improved translation quality and decoding efficiency versus pivot-based baselines on Europarl and WMT datasets.

Proposed method

• Assume that parallel sentences have close probabilities of generating a sentence in a third language to enable direct source-to-target learning without parallel x-y data.
• Train a fixed pivot-to-target teacher model on D_{z,y} and guide the student model on D_{x,z} via sentence-level KL divergence (J_SENT) or word-level KL divergence (J_WORD).
• Sentence-Level Teaching objective minimizes KL(P(y|z;θ̂_{z→y}) || P(y|x;θ_{x→y})) over D_{x,z}.
• Word-Level Teaching objective minimizes KL(P(y|z,y_<j;θ̂_{z→y}) || P(y|x,y_<j;θ_{x→y})) summed over y in the vocabulary and positions j in y.
• Approximate gradients by restricting target space (e.g., beam/mode) to handle intractable full-space KL optimization.
• Use standard NMT decoding with P(y|x;θ_{x→y}) for translation after training.

Experimental results

Research questions

• RQ1 Can a pivot-to-target NMT model teach a source-to-target model without x–y parallel data?
• RQ2 Do sentence-level and word-level teaching approaches effectively transfer knowledge from the teacher to the student?
• RQ3 How does the proposed method compare to pivot-based and multilingual zero-resource approaches in BLEU and decoding efficiency?
• RQ4 Is the teacher-student framework robust across Europarl and WMT datasets and multiple language pairs?
• RQ5 What is the impact of approximate inference (k-best, mode, sampling) on training and translation quality?

Key findings

• The word-level teaching with sampling outperforms pivot-based zero-resource baselines by up to +3.29 BLEU on Es-Fr and +3.24 BLEU on De-Fr in Europarl.
• The sentence-level beam-based teaching also improves over pivot baselines, though with higher training time.
• Across Europarl, both sent-beam and word-sampling surpass pivot-based methods, with word-sampling achieving the best BLEU (Es-Fr: 27.03 on test; De-Fr: 25.15 on test in certain setups).
• On the large-scale WMT Es-Fr task, the word-sampling method attains notable gains over pivot and many-to-one baselines (e.g., +3.46 BLEU over pivot on Newstest2012).
• The proposed approach can outperform a likelihood-based method that uses a small source-target parallel corpus, under zero-resource settings.
• Assumptions grounding the method (sentence- and word-level closeness of P(y|x) and P(y|z)) are empirically validated by decreasing KL divergences during training.