[Paper Review] AdMix: A Mixed Sample Data Augmentation Method for Neural Machine Translation

Chang Jin, Shigui Qiu+2 more|arXiv (Cornell University)|May 10, 2022

Natural Language Processing Techniques4 citations

TL;DR

AdMix proposes a novel data augmentation method for neural machine translation that enhances model performance by softly mixing original sentences with discretely noised variants (via word replacement, dropping, or swapping), followed by residual blending using Beta and Dirichlet distributions. It achieves 1.0–2.7 BLEU point gains over strong Transformer baselines and improves robustness to input perturbations.

ABSTRACT

In Neural Machine Translation (NMT), data augmentation methods such as back-translation have proven their effectiveness in improving translation performance. In this paper, we propose a novel data augmentation approach for NMT, which is independent of any additional training data. Our approach, AdMix, consists of two parts: 1) introduce faint discrete noise (word replacement, word dropping, word swapping) into the original sentence pairs to form augmented samples; 2) generate new synthetic training data by softly mixing the augmented samples with their original samples in training corpus. Experiments on three translation datasets of different scales show that AdMix achieves signifi cant improvements (1.0 to 2.7 BLEU points) over strong Transformer baseline. When combined with other data augmentation techniques (e.g., back-translation), our approach can obtain further improvements.

Motivation & Objective

• To address the limited diversity and semantic inconsistency in existing discrete noising-based data augmentation methods for NMT.
• To improve model generalization and robustness by generating synthetic training samples that preserve semantic correspondence between source and target sentences.
• To develop a data augmentation method independent of additional monolingual data or complex training procedures.
• To explore the synergy between AdMix and other augmentation techniques like back-translation.

Proposed method

• AdMix introduces faint discrete noise (word replacement, dropping, swapping) into original sentence pairs to generate augmented samples.
• It applies soft mixing by sampling from a Dirichlet(α, ..., α) distribution to combine augmented and original samples.
• A residual connection is used to further blend the mixed samples with the original via a Beta(β, β) distribution.
• The final synthetic samples are formed through two-stage convex combination, preserving semantic consistency.
• The method is designed to maintain alignment between source and target sentences during augmentation.
• Hyperparameters such as noise fraction γ and mixing strength α, β are tuned for optimal performance.

Experimental results

Research questions

• RQ1 Can discrete noising combined with soft mixing improve NMT performance without requiring additional monolingual data?
• RQ2 Does AdMix enhance model robustness to input perturbations compared to standard noising or baseline models?
• RQ3 How does AdMix perform across diverse translation tasks of varying scale and language pairs?
• RQ4 Can AdMix be effectively combined with other data augmentation techniques like back-translation to yield further gains?

Key findings

• AdMix achieves 1.0 to 2.7 BLEU point improvements over strong Transformer baselines on IWSLT14 De-En, LDC Zh-En, and WMT14 En-De translation tasks.
• On the German-English validation set, AdMix achieves 38.28 BLEU with 0.1 noise fraction, outperforming all baselines including SeqMix and Swap.
• AdMix maintains stable performance even at higher noise levels (up to 0.4), with a 32.58 BLEU score after three operations on noisy inputs.
• The model trained with AdMix shows superior robustness, decreasing only from 38.28 to 32.58 BLEU over three operations, while the vanilla Transformer drops from 35.82 to 29.75.
• When combined with back-translation, AdMix yields an additional 0.36 BLEU improvement over using AdMix alone, demonstrating synergistic gains.
• The method consistently outperforms all baselines across all noise levels and robustness evaluation settings, confirming its effectiveness and stability.