[Paper Review] Rethinking Perturbations in Encoder-Decoders for Fast Training

Sho Takase, Shun Kiyono|arXiv (Cornell University)|Apr 5, 2021

Natural Language Processing Techniques44 references4 citations

TL;DR

This paper evaluates the computational efficiency and performance of various perturbation techniques in neural encoder-decoder models for sequence-to-sequence tasks. It finds that simple methods like word dropout and random token replacement achieve comparable or better performance than complex techniques such as scheduled sampling and adversarial perturbations—while being significantly faster, making them superior choices for time-constrained training scenarios.

ABSTRACT

We often use perturbations to regularize neural models. For neural encoder-decoders, previous studies applied the scheduled sampling (Bengio et al., 2015) and adversarial perturbations (Sato et al., 2019) as perturbations but these methods require considerable computational time. Thus, this study addresses the question of whether these approaches are efficient enough for training time. We compare several perturbations in sequence-to-sequence problems with respect to computational time. Experimental results show that the simple techniques such as word dropout (Gal and Ghahramani, 2016) and random replacement of input tokens achieve comparable (or better) scores to the recently proposed perturbations, even though these simple methods are faster. Our code is publicly available at https://github.com/takase/rethink_perturbations.

Motivation & Objective

• To evaluate the computational cost and effectiveness of recent perturbation methods in sequence-to-sequence models.
• To investigate whether complex perturbations like scheduled sampling and adversarial training are truly necessary given their high computational overhead.
• To assess whether simpler, faster perturbation techniques can achieve comparable performance to state-of-the-art methods.
• To provide empirical evidence that simple perturbations can serve as strong, efficient baselines for future research.
• To encourage the use of lightweight perturbations as default baselines in sequence-to-sequence training to reduce training costs.

Proposed method

• The study compares three main perturbation types: word replacement (e.g., scheduled sampling), word dropout (randomly masking tokens), and adversarial perturbations (gradient-based perturbations in embedding space).
• For word replacement, the authors evaluate both uniform (Rep(Uni)) and simple sampling (Rep(Sim)) strategies, where tokens are replaced with randomly sampled ones from the vocabulary.
• Word dropout is applied by randomly masking input tokens with a <UNK> token during training, a technique previously used in language modeling.
• Adversarial perturbations are implemented using virtual adversarial training (VAT), where perturbations are computed via gradient ascent in the embedding space to maximize loss.
• Experiments are conducted on machine translation (WMT English-German), grammatical error correction (BEA), and abstractive summarization (CNN/DM), using Transformer-based models.
• All methods are evaluated under identical training conditions to ensure fair comparison, with training time and model performance (BLEU, BLEU-2, BERTScore) measured.

Experimental results

Research questions

• RQ1 Are recent, complex perturbation methods like scheduled sampling and adversarial training computationally efficient enough for practical use?
• RQ2 Can simpler, faster perturbation techniques such as word dropout and random token replacement achieve comparable performance to advanced methods?
• RQ3 How does the training time of different perturbation strategies compare when achieving similar model performance?
• RQ4 Does the effectiveness of simple perturbations vary across different sequence-to-sequence tasks like translation, summarization, and grammatical error correction?
• RQ5 Can simple perturbations serve as strong, efficient baselines for future research in neural encoder-decoder models?

Key findings

• Word dropout (WDrop) and random replacement (Rep(Uni) + WDrop) achieved BLEU scores comparable to adversarial perturbations (Adv) on the WMT English-German translation task, even when training time was constrained.
• In the low-resource setting, adversarial perturbations achieved high BLEU scores, but simple methods matched them closely when given similar training time.
• The Rep(Sim) method showed superior robustness compared to other perturbation strategies, indicating its effectiveness in generalization.
• Simple perturbation techniques like word dropout and random replacement were significantly faster than scheduled sampling and adversarial training, reducing training overhead.
• On the BEA grammatical error correction dataset, Rep(Uni) + WDrop and Rep(Sim) + WDrop achieved performance on par with Adv, confirming the effectiveness of simple methods beyond translation.
• The authors observed that the performance of the baseline model without perturbations was slightly worse than prior work, likely due to training randomness, but all perturbation methods improved performance over this baseline.