[Paper Review] Sockeye: A Toolkit for Neural Machine Translation

Felix Hieber, Tobias Domhan+5 more|arXiv (Cornell University)|Dec 15, 2017

Natural Language Processing Techniques2 references194 citations

TL;DR

SOCKEYE is an open-source Python MXNet-based NMT toolkit that implements three major architectures (RNN with attention, Transformer, and ConvSeq2Seq), with extensive training/inference features and benchmarks against other toolkits on WMT data.

ABSTRACT

We describe Sockeye (version 1.12), an open-source sequence-to-sequence toolkit for Neural Machine Translation (NMT). Sockeye is a production-ready framework for training and applying models as well as an experimental platform for researchers. Written in Python and built on MXNet, the toolkit offers scalable training and inference for the three most prominent encoder-decoder architectures: attentional recurrent neural networks, self-attentional transformers, and fully convolutional networks. Sockeye also supports a wide range of optimizers, normalization and regularization techniques, and inference improvements from current NMT literature. Users can easily run standard training recipes, explore different model settings, and incorporate new ideas. In this paper, we highlight Sockeye's features and benchmark it against other NMT toolkits on two language arcs from the 2017 Conference on Machine Translation (WMT): English-German and Latvian-English. We report competitive BLEU scores across all three architectures, including an overall best score for Sockeye's transformer implementation. To facilitate further comparison, we release all system outputs and training scripts used in our experiments. The Sockeye toolkit is free software released under the Apache 2.0 license.

Motivation & Objective

• Provide a production-ready, extensible NMT toolkit with support for the three main encoder-decoder architectures.
• Benchmark SOCKEYE against other NMT toolkits on standard WMT datasets to assess translation quality and speed.
• Describe key modeling, training, and inference features that enable researchers to easily train, compare, and extend NMT models.
• Release reproducible system outputs and training scripts to facilitate fair comparisons across toolkits.

Proposed method

• Implementations of three major NMT architectures: stacked RNN with attention, self-attentional Transformer, and Fully Convolutional networks.
• Integration of layer/weight normalization, RNN attention variants, and coverage models to improve training stability and translation quality.
• Comprehensive training features including optimizers (Adam, Eve), learning rate schedules, regularization, and multi-GPU support.
• Efficient inference with beam search, ensemble decoding, batch decoding, vocabulary selection, and attention visualization.
• Evaluation against datasets (EN→DE, LV→EN) using BLEU, with comparison to OpenNMT, Marian, NEMATUS, NeuralMonkey, and others.
• Open-source release of models, outputs, and training scripts for reproducible research.

Experimental results

Research questions

• RQ1 How do the three prominent NMT architectures (RNN with attention, Transformer, ConvSeq2Seq) compare in translation quality and speed within a single toolkit environment?
• RQ2 What training, regularization, and inference techniques best support production-ready NMT across architectures?
• RQ3 How does SOCKEYE perform relative to other open-source toolkits on standard WMT datasets (EN→DE, LV→EN) in terms of BLEU and efficiency?
• RQ4 Can researchers easily train, compare, and extend models using SOCKEYE with reproducible outputs and scripts?

Key findings

• SOCKEYE achieves competitive BLEU scores with its RNN model comparable to the best-performing toolkit in the comparisons.
• The Transformer implementation in SOCKEYE yields the best overall BLEU among the evaluated architectures.
• The ConvSeq2Seq model in SOCKEYE performs best in its class on the benchmarks.
• SOCKEYE demonstrates competitive or superior decoding speeds in various configurations, with vocabulary selection providing substantial speedups.
• The authors release all system outputs and training scripts to facilitate fair comparisons and reproducibility.