[Paper Review] Multilingual Neural Machine Translation with Task-Specific Attention

Graeme Blackwood, Miguel Ballesteros+1 more|arXiv (Cornell University)|Jun 8, 2018

Natural Language Processing TechniquesComputer Science21 references53 citations

TL;DR

The paper introduces task-specific attention variants for multilingual NMT to improve translation quality across multiple language directions, including zero-shot, with only a small parameter overhead.

ABSTRACT

Multilingual machine translation addresses the task of translating between multiple source and target languages. We propose task-specific attention models, a simple but effective technique for improving the quality of sequence-to-sequence neural multilingual translation. Our approach seeks to retain as much of the parameter sharing generalization of NMT models as possible, while still allowing for language-specific specialization of the attention model to a particular language-pair or task. Our experiments on four languages of the Europarl corpus show that using a target-specific model of attention provides consistent gains in translation quality for all possible translation directions, compared to a model in which all parameters are shared. We observe improved translation quality even in the (extreme) low-resource zero-shot translation directions for which the model never saw explicitly paired parallel data.

Motivation & Objective

• Improve translation quality in fully multilingual NMT by specializing attention per task while sharing other parameters.
• Maintain parameter sharing to leverage cross-language generalization.
• Enhance zero-shot and low-resource directions without requiring separate models for every language pair.

Proposed method

• Extend attentional NMT with three task-specific attention variants: target-specific, source-specific, and paired attention.
• Condition attention weights and biases on the chosen task (language or language pair) during training and decoding.
• Augment encoder inputs with task tokens to indicate the desired target language.
• Train with a merged Europarl dataset across multiple language directions using shared embeddings and a common subword vocabulary (80k BPE).
• Use a single recurrent encoder/decoder with attention, and add a small number of task-specific parameters (one attention weight matrix and one bias per task).
• Evaluate with BLEU on in-domain dev/test sets and out-of-domain News Commentary, averaging results over five random seeds.

Experimental results

Research questions

• RQ1 Does target-specific attention improve multilingual MT compared to fully shared attention?
• RQ2 How do the three task-specific attention variants perform across all translation directions, including zero-shot?
• RQ3 What is the impact on parameter count and training efficiency?
• RQ4 Are improvements robust to out-of-domain data?
• RQ5 How does attention visualization differ between shared and target-specific variants?

Key findings

• Target-specific attention yields consistent BLEU gains of roughly +0.5 to +0.9 over shared-attention multilingual models across six directions.
• Zero-shot translations improve by about +1.0 to +1.5 BLEU with target-specific attention, while source-specific attention is less effective for zero-shot.
• Paired attention shows little improvement over shared attention, likely due to fewer data per task and no explicit zero-shot handling.
• Overall multilingual models with target-specific attention reduce degradation seen in multilingual setups compared to fully shared attention, and perform robustly on out-of-domain data, with gains of +0.6 to +1.2 BLEU on nc-dev2007/nc-devtest2007.
• The approach introduces only a small parameter overhead (target-specific model adds one d×d weight matrix and d biases per target language, about 1.2% increase for four targets).