[Paper Review] Semi-Autoregressive Training Improves Mask-Predict Decoding

Marjan Ghazvininejad, Omer Levy+1 more|arXiv (Cornell University)|Jan 23, 2020

Topic Modeling15 references48 citations

TL;DR

SMART training improves conditional masked language models for mask-predict decoding, closing most of the gap to autoregressive models and boosting translation quality.

ABSTRACT

The recently proposed mask-predict decoding algorithm has narrowed the performance gap between semi-autoregressive machine translation models and the traditional left-to-right approach. We introduce a new training method for conditional masked language models, SMART, which mimics the semi-autoregressive behavior of mask-predict, producing training examples that contain model predictions as part of their inputs. Models trained with SMART produce higher-quality translations when using mask-predict decoding, effectively closing the remaining performance gap with fully autoregressive models.

Motivation & Objective

• Motivate and address the mismatch between training and semi-autoregressive mask-predict decoding.
• Propose SMART training to better reflect prediction-based inputs during training.
• Evaluate SMART on standard MT benchmarks and compare with NART and autoregressive baselines.
• Analyze how training choices (e.g., predicting all tokens, forward passes) affect performance and robustness.

Proposed method

• Start with a gold target sequence and mask a random subset to create Y_gold.
• Predict the entire sequence to obtain Y_pred from partially observed Y_gold_obs.
• Mask a new random subset from Y_pred to create the training input Y_pred_obs and train to predict the gold Y_gold.
• Train with cross-entropy over all tokens to allow correcting mistakes in observed predictions.
• Optionally modify the mask-predict predict step to predict all tokens (not only masked) for robustness.

Experimental results

Research questions

• RQ1 Does SMART training improve translation quality when using mask-predict decoding compared to original NART training?
• RQ2 How close do SMART-trained models get to fully autoregressive baselines on standard MT benchmarks?
• RQ3 What training variabilities (e.g., predicting all tokens vs. masked tokens, number of forward passes) affect performance?
• RQ4 How does SMART perform across language pairs and decoding iterations (T) in mask-predict?
• RQ5 What is the impact of training example difficulty (gold masking ratio) on learning?

Key findings

• SMART yields higher BLEU than NART across benchmarks, averaging +0.71 BLEU.
• With 10 decoding iterations, SMART reaches 27.65 BLEU on WMT’14 EN-DE, closely matching 27.75 BLEU of a strong autoregressive baseline.
• SMART at 10 iterations with mask-predict achieves 31.27 (DE-EN) and 34.06 (EN-ZH) on WMT’17, nearing autoregressive performance.
• Training with SMART and predicting all tokens during the predict step provides a ~0.40 BLEU gain on development set with 4 iterations.
• Increasing iterations generally narrows gaps to autoregressive models, with some benchmarks showing equal or near-equal performance.