[Paper Review] Revisiting Self-Training for Neural Sequence Generation

Junxian He, Jiatao Gu+2 more|arXiv (Cornell University)|Sep 30, 2019

Topic ModelingComputer Science36 references139 citations

TL;DR

The paper revisits self-training for neural sequence generation and shows that adding dropout and input noise (noisy self-training) substantially improves semi-supervised learning for MT and summarization, sometimes outperforming back-translation especially in low-resource settings.

ABSTRACT

Self-training is one of the earliest and simplest semi-supervised methods. The key idea is to augment the original labeled dataset with unlabeled data paired with the model's prediction (i.e. the pseudo-parallel data). While self-training has been extensively studied on classification problems, in complex sequence generation tasks (e.g. machine translation) it is still unclear how self-training works due to the compositionality of the target space. In this work, we first empirically show that self-training is able to decently improve the supervised baseline on neural sequence generation tasks. Through careful examination of the performance gains, we find that the perturbation on the hidden states (i.e. dropout) is critical for self-training to benefit from the pseudo-parallel data, which acts as a regularizer and forces the model to yield close predictions for similar unlabeled inputs. Such effect helps the model correct some incorrect predictions on unlabeled data. To further encourage this mechanism, we propose to inject noise to the input space, resulting in a "noisy" version of self-training. Empirical study on standard machine translation and text summarization benchmarks shows that noisy self-training is able to effectively utilize unlabeled data and improve the performance of the supervised baseline by a large margin.

Motivation & Objective

• Evaluate how self-training performs on neural sequence generation tasks (e.g., machine translation and text summarization).
• Identify why self-training helps or fails, and determine the key ingredients that drive gains.
• Propose and validate a noisy self-training variant to better utilize unlabeled data.
• Provide practical guidance on when and how self-training yields improvements in seq2seq models.

Proposed method

• Formalize classic self-training for conditional sequence generation with a base model trained on L and pseudo-parallel data S from unlabeled U.
• Show that training a new model on S ∪ L (train from pseudo data first, then fine-tune on real data) can improve over the baseline.
• Experiment with decoding strategies (beam search vs sampling) to generate pseudo targets.
• Introduce dropout during pseudo-training as a crucial regularizer that enables the model to yield similar predictions for similar inputs.
• Propose noisy self-training by injecting input perturbations (noise) in addition to dropout to further smooth the input-output mapping.
• Empirically validate on MT (WMT 2014 En–De, FloRes En–Ne) and Gigaword summarization across high- and low-resource settings.

Experimental results

Research questions

• RQ1 How does self-training perform on neural sequence generation tasks like machine translation and summarization?
• RQ2 What factors contribute to gains from self-training, and are ‘bad’ pseudo targets a problem?
• RQ3 Can adding noise to the input and internal representations (noisy self-training) improve utilization of unlabeled data across tasks and resource settings?

Key findings

• Self-training yields significant improvements over supervised baselines on neural sequence generation tasks.
• Dropout on the teacher/model during pseudo-training is the key factor driving most gains, acting as a regularizer to encourage consistent predictions on unlabeled data.
• Beam-search decoding contributes modestly to gains, but dropout-driven perturbations are the primary source of improvement.
• Noisy self-training, which adds input perturbations to the pseudo-training process, further boosts performance across machine translation and text summarization benchmarks.
• On WMT English–German and FloRes English–Nepali, noisy self-training outperforms baselines by 1–6 BLEU points in many settings; it also matches or exceeds back-translation in some cases, especially with limited parallel data.
• In text summarization (Gigaword), noisy self-training often surpasses back-translation with smaller parallel data and approaches state-of-the-art results in full-data pretraining scenarios.