[Paper Review] Learning to Discriminate Noises for Incorporating External Information in Neural Machine Translation

Zaixiang Zheng, Shujian Huang+4 more|arXiv (Cornell University)|Oct 24, 2018

Natural Language Processing Techniques35 references4 citations

TL;DR

This paper proposes a noise-discriminative framework for enhancing Neural Machine Translation (NMT) by jointly identifying global and local noises in external information—such as SMT outputs, dictionaries, or human corrections—without requiring labeled data. Trained on synthetic data from parallel corpora, the model significantly improves translation quality, especially under high noise, by filtering irrelevant words during decoding.

ABSTRACT

Previous studies show that incorporating external information could improve the translation quality of Neural Machine Translation (NMT) systems. However, there are inevitably noises in the external information, severely reducing the benefit that the existing methods could receive from the incorporation. To tackle the problem, this study pays special attention to the discrimination of the noises during the incorporation. We argue that there exist two kinds of noise in this external information, i.e. global noise and local noise, which affect the translations for the whole sentence and for some specific words, respectively. Accordingly, we propose a general framework that learns to jointly discriminate both the global and local noises, so that the external information could be better leveraged. Our model is trained on the dataset derived from the original parallel corpus without any external labeled data or annotation. Experimental results in various real-world scenarios, language pairs, and neural architectures indicate that discriminating noises contributes to significant improvements in translation quality by being able to better incorporate the external information, even in very noisy conditions.

Motivation & Objective

• To address the challenge of noisy external information in Neural Machine Translation (NMT), which undermines the benefits of incorporating such data.
• To identify two distinct types of noise: global noise (irrelevant to the entire sentence) and local noise (irrelevant at specific translation steps).
• To develop a generalizable framework that discriminates both noise types without requiring external annotations or task-specific tuning.
• To demonstrate that noise discrimination leads to consistent improvements across diverse real-world scenarios and language pairs.

Proposed method

• The framework employs two separate discriminators: a global word discriminator to assess the relevance of external words to the entire sentence, and a local word discriminator to evaluate relevance at each decoding step.
• External information is represented as a set of words (external words), generalizing across sources like SMT outputs, bilingual dictionaries, or human corrections.
• The model is trained on synthetic data generated by sampling words from parallel sentences, requiring no human-annotated noise labels.
• The training process uses the original parallel corpus to create noisy yet realistic external word sets, enabling self-supervised learning.
• During inference, the discriminators filter out noisy words before they influence the NMT decoder’s attention or output generation.
• The framework is applied uniformly across different neural architectures and real-world external data sources without retraining or adaptation.

Experimental results

Research questions

• RQ1 How do global and local noises in external information affect NMT translation quality?
• RQ2 Can a unified framework effectively discriminate both global and local noise without relying on external annotations?
• RQ3 Does noise discrimination lead to measurable improvements in translation performance, especially in high-noise scenarios?
• RQ4 Can a model trained on synthetic data generalize across diverse external information sources without task-specific fine-tuning?

Key findings

• Noise in external information significantly hinders the performance gains from incorporating such data in NMT systems.
• The proposed noise-discriminative framework achieves a BLEU score of 35.71 on the SMT scenario, outperforming the baseline (33.28) and showing the largest gain in noisy conditions.
• In the Lex scenario, the model improves from 32.57 (baseline) to 34.10 BLEU, demonstrating effectiveness even with word-level translation tables.
• The model generalizes well across four real-world scenarios—SMT, lexical tables, dictionaries, and bag-of-words predictions—without retraining or tuning.
• Case studies show that the model successfully corrects NMT errors (e.g., replacing "working" with "moving") using external word inputs, even with minimal human correction.
• The framework maintains strong performance across different neural architectures and language pairs, confirming its robustness and transferability.