[Paper Review] Massive Exploration of Neural Machine Translation Architectures

Denny Britz, Anna Goldie+2 more|arXiv (Cornell University)|Mar 11, 2017

Natural Language Processing TechniquesComputer Science21 references70 citations

TL;DR

This work conducts a large-scale hyperparameter sweep over neural machine translation architectures, revealing which architectural choices most influence BLEU, perplexity, and training efficiency, and releasing an open-source NMT framework for reproducible research.

ABSTRACT

Neural Machine Translation (NMT) has shown remarkable progress over the past few years with production systems now being deployed to end-users. One major drawback of current architectures is that they are expensive to train, typically requiring days to weeks of GPU time to converge. This makes exhaustive hyperparameter search, as is commonly done with other neural network architectures, prohibitively expensive. In this work, we present the first large-scale analysis of NMT architecture hyperparameters. We report empirical results and variance numbers for several hundred experimental runs, corresponding to over 250,000 GPU hours on the standard WMT English to German translation task. Our experiments lead to novel insights and practical advice for building and extending NMT architectures. As part of this contribution, we release an open-source NMT framework that enables researchers to easily experiment with novel techniques and reproduce state of the art results.

Motivation & Objective

• Identify how architectural choices affect NMT performance, training efficiency, and stability.
• Provide actionable guidance for building and extending NMT architectures.
• Quantify variability due to initialization and hyperparameters to distinguish signal from noise.
• Promote reproducibility by releasing an open-source NMT framework and data preprocessing scripts.

Proposed method

• Perform a comprehensive hyperparameter sweep over encoder/decoder depth, RNN cell type (LSTM/GRU/vanilla), embedding size, residual connections, bidirectionality, attention type and dimensionality.
• Use an encoder–decoder with attention baseline and evaluate on WMT’15 English→German with 4.5M sentence pairs.
• Train with Adam, constant learning rate, 2.5M steps, and 4x replications; evaluate with BLEU on newstest2013/2014/2015.
• Systematically vary one hyperparameter at a time while keeping others fixed to isolate effects.
• Release an open-source TensorFlow-based framework and all configuration/scripts for reproducibility.

Experimental results

Research questions

• RQ1 What architectural choices in NMT most affect translation quality (BLEU) and convergence behavior on the WMT English→German task?
• RQ2 How do embedding size, encoder/decoder depth, RNN cell type, and attention mechanism influence performance and training stability?
• RQ3 To what extent do beam search settings and length penalties impact final translation quality?
• RQ4 Are larger, more complex architectures consistently better, or do optimization challenges limit gains?
• RQ5 How much do random initialization and hyperparameter variation contribute to observed performance differences?

Key findings

• 2048-dimensional embeddings yield the best BLEU broadly but offer only marginal gains over smaller embeddings.
• LSTM cells outperform GRU and vanilla decoders do poorly, indicating gating is beneficial for the decoder.
• Deep encoders beyond 2 layers are unstable unless using dense residual connections; deep decoders benefit modestly from depth with residuals.
• Bidirectional encoders generally outperform unidirectional ones, with reversed input sometimes improving results.
• Parameterized additive attention slightly outperforms multiplicative attention, and attention dimensionality has limited impact.
• Well-tuned beam search with length penalty is crucial; beam width around 5–10 with LP=1.0 performs well, while very large beams can hurt results.