[Paper Review] Word-based Domain Adaptation for Neural Machine Translation

Yan Shen, Leonard Dahlmann+3 more|arXiv (Cornell University)|Jun 7, 2019

Natural Language Processing Techniques22 references4 citations

TL;DR

This paper proposes a word-level domain adaptation method for neural machine translation (NMT) in e-commerce settings, using logarithmic probability differences from domain-specific and general language models to assign word weights. After smoothing and binary quantization, these weights improve training by emphasizing in-domain-like words in out-of-domain data, yielding up to 2.11 BLEU and 1.59 TER gains over baseline methods, with further gains when combined with fine-tuning.

ABSTRACT

In this paper, we empirically investigate applying word-level weights to adapt neural machine translation to e-commerce domains, where small e-commerce datasets and large out-of-domain datasets are available. In order to mine in-domain like words in the out-of-domain datasets, we compute word weights by using a domain-specific and a non-domain-specific language model followed by smoothing and binary quantization. The baseline model is trained on mixed in-domain and out-of-domain datasets. Experimental results on English to Chinese e-commerce domain translation show that compared to continuing training without word weights, it improves MT quality by up to 2.11% BLEU absolute and 1.59% TER. We have also trained models using fine-tuning on the in-domain data. Pre-training a model with word weights improves fine-tuning up to 1.24% BLEU absolute and 1.64% TER, respectively.

Motivation & Objective

• To address low-resource e-commerce domain translation by leveraging small in-domain and large out-of-domain datasets.
• To improve NMT performance through word-level weighting instead of sentence- or chunk-level weighting.
• To reduce noise in word score assignments using smoothing techniques before thresholding.
• To demonstrate that pre-training with word weights enhances subsequent fine-tuning on in-domain data.

Proposed method

• Compute word scores using the log difference between probabilities from a domain-specific language model (LM) and a non-domain-specific LM.
• Apply smoothing via a weighted moving average filter over neighboring words to reduce noise in word scores.
• Perform binary quantization of smoothed scores to produce word weights of 0 or 1, selecting in-domain-like words.
• Train the NMT model using a modified objective that scales the cross-entropy loss by word weights.
• Use the word-weighted model as a pre-trained base before fine-tuning on in-domain data.
• Evaluate performance using BLEU and TER on an English-to-Chinese e-commerce translation benchmark.

Experimental results

Research questions

• RQ1 Can word-level weighting based on language model probability differences improve NMT performance in low-resource e-commerce domains?
• RQ2 How does smoothing word scores affect the quality of selected in-domain-like words in out-of-domain data?
• RQ3 Does pre-training with word weights lead to better fine-tuning performance than direct fine-tuning on in-domain data?
• RQ4 How does word-level weighting compare to sentence-level or chunk-level weighting in domain adaptation?

Key findings

• Smoothing word scores before thresholding significantly improves translation quality, with BLEU increasing from 21.38% to 26.14% and TER decreasing from 66.25% to 60.34%.
• Continuing training with word weights improves BLEU by 2.11 absolute and TER by 1.59 absolute compared to training without word weights.
• Fine-tuning after pre-training with word weights improves BLEU by 1.24 absolute and TER by 1.64 absolute compared to direct fine-tuning.
• The longest consecutive words (LCW) method effectively preserves meaningful in-domain segments like "防/溢出/喷口/内" while filtering out isolated, noisy words such as "空气".
• The method outperforms random word masking and non-smoothed word weighting, confirming the necessity of noise reduction in word score assignment.
• The approach generalizes instance weighting by enabling per-word selection based on domain similarity, improving data efficiency in low-resource settings.