[Paper Review] Mixture Models for Diverse Machine Translation: Tricks of the Trade

Tianxiao Shen, Myle Ott+2 more|arXiv (Cornell University)|Feb 20, 2019

Topic ModelingComputer Science61 citations

TL;DR

The paper provides a comprehensive empirical study of conditional mixture models (mixture of experts) for diverse neural machine translation, identifying key design choices and training tricks that yield a favorable quality-diversity trade-off compared to variational NMT and diverse decoding baselines.

ABSTRACT

Mixture models trained via EM are among the simplest, most widely used and well understood latent variable models in the machine learning literature. Surprisingly, these models have been hardly explored in text generation applications such as machine translation. In principle, they provide a latent variable to control generation and produce a diverse set of hypotheses. In practice, however, mixture models are prone to degeneracies---often only one component gets trained or the latent variable is simply ignored. We find that disabling dropout noise in responsibility computation is critical to successful training. In addition, the design choices of parameterization, prior distribution, hard versus soft EM and online versus offline assignment can dramatically affect model performance. We develop an evaluation protocol to assess both quality and diversity of generations against multiple references, and provide an extensive empirical study of several mixture model variants. Our analysis shows that certain types of mixture models are more robust and offer the best trade-off between translation quality and diversity compared to variational models and diverse decoding approaches.\footnote{Code to reproduce the results in this paper is available at \url{https://github.com/pytorch/fairseq}}

Motivation & Objective

• Motivate the need to model multi-modal translation outputs with diverse hypotheses.
• Evaluate a wide range of mixture-model design choices for MT (hard/soft EM, priors, parameter sharing).
• Develop and apply an evaluation protocol that jointly measures translation quality and diversity using multiple references.
• Identify practical training tricks that mitigate degeneracies and promote meaningful specialization of experts.

Proposed method

• Use conditional mixture models (Mixture of Experts) where a latent variable z selects among K experts to generate y given x.
• Investigate hard vs soft EM, learned vs uniform priors, and online vs offline responsibility updates.
• Compare independent vs shared parameterization of experts and analyze the impact of regularization dropout on responsibility computation.
• Train and evaluate on large-scale MT benchmarks (WMT En-De, En-Fr, Zh-En) using Transformer architectures in Fairseq.
• Propose a simple decoding strategy: generate K hypotheses by enumerating z and greedy decoding per expert.

Experimental results

Research questions

• RQ1 Can mixture-of-experts MT models produce diverse and high-quality translations simultaneously?
• RQ2 How do design choices (EM type, prior, parameter sharing, update schedule, regularization) affect training degeneracies and performance?
• RQ3 Which configurations yield the best quality-diversity trade-off compared to baselines like variational NMT and diverse decoding?
• RQ4 What practical guidelines emerge for training robust, diverse MT models with mixture components?

Key findings

• Certain mixture-model variants robustly balance quality and diversity and approach human performance in multi-reference settings.
• Disabling dropout in the E-step is crucial to prevent degeneracies where only one expert is trained or the latent variable is ignored.
• Hard EM with uniform priors and online responsibility updates (hMup, online-shared) offers strong diversity with a favorable quality–diversity trade-off.
• Independent parameterization with offline responsibility updates can yield higher diversity but may reduce quality; shared parameters with online updates strike a robust balance.
• Across three large MT benchmarks, the Mixture of Experts model (hMup) outperforms variational NMT and diverse decoding baselines in overall quality-diversity trade-offs.