[Paper Review] Robust Neural Machine Translation with Joint Textual and Phonetic Embedding

Hairong Liu, Mingbo Ma+3 more|arXiv (Cornell University)|Oct 15, 2018

Natural Language Processing Techniques23 references4 citations

TL;DR

This paper proposes a joint textual and phonetic embedding approach to enhance neural machine translation (NMT) robustness against homophone noises—where words are replaced by phonetically similar alternatives. By emphasizing phonetic information (with β ≈ 0.95) and augmenting training data with synthetic homophone errors, the method significantly improves robustness on noisy inputs and surprisingly boosts performance on clean test sets, outperforming standard NMT models in both scenarios.

ABSTRACT

Neural machine translation (NMT) is notoriously sensitive to noises, but noises are almost inevitable in practice. One special kind of noise is the homophone noise, where words are replaced by other words with similar pronunciations. We propose to improve the robustness of NMT to homophone noises by 1) jointly embedding both textual and phonetic information of source sentences, and 2) augmenting the training dataset with homophone noises. Interestingly, to achieve better translation quality and more robustness, we found that most (though not all) weights should be put on the phonetic rather than textual information. Experiments show that our method not only significantly improves the robustness of NMT to homophone noises, but also surprisingly improves the translation quality on some clean test sets.

Motivation & Objective

• To address the vulnerability of NMT models to homophone noise, where phonetically similar words are substituted, leading to incorrect translations.
• To improve robustness without sacrificing performance on clean test sets, which is often overlooked in robust NMT research.
• To investigate whether phonetic information can serve as a more stable signal than textual input under homophone noise.
• To evaluate the effectiveness of data augmentation using synthetic homophone errors in enhancing model generalization.

Proposed method

• Jointly embed words and their phonetic pronunciations (e.g., Pinyin for Chinese) using learnable embeddings for both textual and phonetic units.
• Combine word and phonetic embeddings via a weighted average: π([a,ψ(a)]) = (1−β)π(a) + βπ(ψ(a)), where β controls the balance between textual and phonetic signals.
• Use β ≈ 0.95 in experiments, indicating that phonetic information dominates the final representation for optimal robustness.
• Apply data augmentation by randomly replacing source words with their homophones in 40% of training sentence pairs, increasing training data to ~2.8M pairs.
• Train transformer-based NMT models with the joint embedding layer and augmented data, evaluating on both clean and noisy test sets.
• Use averaging to compress phonetic unit embeddings into a single vector before combining with word embeddings, avoiding complex RNNs for merging.

Experimental results

Research questions

• RQ1 Can joint textual and phonetic embedding improve NMT robustness to homophone noise in real-world input scenarios?
• RQ2 Does relying more on phonetic information than textual input lead to better robustness and generalization in NMT?
• RQ3 To what extent does data augmentation with synthetic homophone errors improve model robustness and performance on clean data?
• RQ4 How does the balance between textual and phonetic embeddings (controlled by β) affect translation quality and noise resilience?

Key findings

• The model with β = 0.95, emphasizing phonetic information, achieved the best performance, indicating that phonetic signals are more robust than textual ones under homophone noise.
• On the NoisySet1 and NoisySet2 test sets, the proposed method achieved 45.71 and 42.66 BLEU scores respectively, significantly outperforming the baseline Transformer (41.33 and 37.11).
• After data augmentation, the model’s BLEU score on clean test sets (NIST 06) improved from 45.97 to 48.06, showing that noisy data helps generalization.
• The method successfully translated a sentence with a homophone substitution (有 → 又) correctly, while the baseline model generated irrelevant output (e.g., 'hpv' instead of 'have').
• Even on clean test sets, the model with joint embeddings and data augmentation achieved higher BLEU scores than the baseline, indicating that robustness training improves overall performance.
• The ablation study confirmed that phonetic information is more critical than textual information for noise robustness, with β close to 1 yielding optimal results.