[Paper Review] Multi-Domain Neural Machine Translation with Word-Level Adaptive Layer-wise Domain Mixing

Haoming Jiang, Liang Chen+2 more|arXiv (Cornell University)|Nov 7, 2019

Natural Language Processing Techniques38 references4 citations

TL;DR

This paper proposes a multi-domain neural machine translation model that uses word-level, adaptive, layer-wise domain mixing to improve both domain-shared and domain-specific knowledge learning. By assigning dynamic domain proportions per word and layer, and combining domain-specific multi-head attention modules via weighted averaging, the model achieves state-of-the-art performance on multiple NMT benchmarks, outperforming existing methods in both multi-domain and single-domain translation tasks.

ABSTRACT

Many multi-domain neural machine translation (NMT) models achieve knowledge transfer by enforcing one encoder to learn shared embedding across domains. However, this design lacks adaptation to individual domains. To overcome this limitation, we propose a novel multi-domain NMT model using individual modules for each domain, on which we apply word-level, adaptive and layer-wise domain mixing. We first observe that words in a sentence are often related to multiple domains. Hence, we assume each word has a domain proportion, which indicates its domain preference. Then word representations are obtained by mixing their embedding in individual domains based on their domain proportions. We show this can be achieved by carefully designing multi-head dot-product attention modules for different domains, and eventually taking weighted averages of their parameters by word-level layer-wise domain proportions. Through this, we can achieve effective domain knowledge sharing, and capture fine-grained domain-specific knowledge as well. Our experiments show that our proposed model outperforms existing ones in several NMT tasks.

Motivation & Objective

• To address the limitation of existing multi-domain NMT models that rely on a single shared encoder, which fails to adapt well to individual domains.
• To enable fine-grained, context-dependent domain knowledge sharing by allowing words to have multiple domain proportions across layers.
• To improve translation performance by capturing both domain-shared and domain-specific representations through adaptive mixing of domain-specific attention modules.
• To demonstrate that word-level, layer-wise domain mixing can outperform standard unified or embedding-based multi-domain NMT approaches.

Proposed method

• The model constructs separate multi-head dot-product attention modules for each domain, enabling domain-specific representation learning.
• For each word, domain proportions are computed per layer, reflecting its contextual preference for different domains.
• Word representations are generated by taking weighted averages of the parameters of domain-specific attention modules, using the word’s layer-wise domain proportions as weights.
• The domain proportion for each word is modeled as a learnable, context-dependent vector that evolves across layers.
• The method integrates with existing domain-aware embedding techniques (e.g., MTL, AdvL, PAdvL) by allowing gradients to flow from the domain proportion module to the embedding space.
• The architecture supports end-to-end training with backpropagation, enabling joint optimization of domain mixing and translation tasks.

Experimental results

Research questions

• RQ1 Can word-level, layer-wise domain mixing improve multi-domain NMT performance by enabling adaptive knowledge sharing across domains?
• RQ2 How does the dynamic assignment of domain proportions per word and layer affect the model’s ability to disambiguate domain-specific word meanings?
• RQ3 To what extent can domain-specific attention modules outperform a single shared encoder in capturing domain-specific linguistic patterns?
• RQ4 Can the proposed mixing mechanism be effectively combined with existing domain-aware embedding methods to further improve translation quality?
• RQ5 How do domain proportions evolve across layers, and what does this reveal about knowledge sharing and specialization in the network?

Key findings

• The proposed model achieves state-of-the-art BLEU scores on multiple multi-domain NMT benchmarks, outperforming existing methods including unified models and embedding-based approaches.
• On the Chinese-to-English translation task, combining domain mixing with PAdvL embedding training improved BLEU scores by up to 0.48 points compared to baseline.
• The decoder’s domain proportions were more skewed than the encoder’s, indicating weaker domain knowledge sharing in the decoder, which limited performance gains in the English-to-German task.
• Word disambiguation was improved through evolving domain proportions: the same word like 'article' could be mapped to different domains (e.g., 'laws' vs. 'media') based on context and layer.
• The model effectively captured domain-specific representations, as shown by the evolution of domain proportions across layers, with words showing distinct domain preferences at different depths.
• Experiments confirmed that the domain mixing mechanism is complementary to domain-aware embedding techniques, and combining them yields further performance gains.