[Paper Review] Findings of the Second Workshop on Neural Machine Translation and Generation

Alexandra Birch, Andrew Finch+3 more|arXiv (Cornell University)|Jun 8, 2018

Natural Language Processing Techniques28 references3 citations

TL;DR

This paper presents findings from the Second Workshop on Neural Machine Translation and Generation (WNMT 2018), focusing on advances in efficient neural machine translation (NMT). It introduces a shared task evaluating systems on accuracy, inference speed, and memory efficiency, with results showing that optimized systems—particularly from the Marian team—achieved superior performance on the Pareto frontier of speed and accuracy, especially on CPU, through model distillation, quantization, and low-level optimizations.

ABSTRACT

This document describes the findings of the Second Workshop on Neural Machine Translation and Generation, held in concert with the annual conference of the Association for Computational Linguistics (ACL 2018). First, we summarize the research trends of papers presented in the proceedings, and note that there is particular interest in linguistic structure, domain adaptation, data augmentation, handling inadequate resources, and analysis of models. Second, we describe the results of the workshop's shared task on efficient neural machine translation, where participants were tasked with creating MT systems that are both accurate and efficient.

Motivation & Objective

• To evaluate and advance the state of the art in neural machine translation (NMT) with a focus on practical efficiency, not just accuracy.
• To establish a shared task that measures both translation quality and computational efficiency, including inference speed and memory usage.
• To identify best practices in model optimization, such as distillation, quantization, and efficient inference, for real-world NMT deployment.
• To analyze the trade-offs between model architecture (e.g., RNN vs. self-attention), hardware (CPU vs. GPU), and efficiency metrics.
• To provide strong baselines and evaluation frameworks for future research on efficient NMT systems.

Proposed method

• Conducted a shared task on efficient NMT, requiring participants to build systems that balance accuracy, inference speed, and memory usage.
• Evaluated systems using BLEU and NIST for accuracy, and measured inference time on CPU and GPU, including model loading overhead.
• Measured memory efficiency via model size on disk, number of parameters, and peak host/GPU memory consumption.
• Used Docker containers to isolate and measure resource usage in controlled environments on Amazon Web Services.
• Employed model distillation to transfer knowledge from large teacher models to smaller, faster student models (e.g., RNN-based or distilled self-attention models).
• Applied low-precision computation (e.g., int8 quantization) and auto-tuning of matrix multiplication kernels to improve inference speed.

Experimental results

Research questions

• RQ1 How do different model architectures (e.g., RNNs vs. self-attention) perform in terms of speed and accuracy on CPU versus GPU?
• RQ2 To what extent can model distillation and quantization improve inference efficiency without sacrificing translation quality?
• RQ3 What are the most effective engineering optimizations for reducing memory and computation costs in NMT systems?
• RQ4 Can data augmentation techniques like back-translation improve efficiency and accuracy in low-resource settings?
• RQ5 What is the trade-off between model size, inference speed, and translation quality in practical deployment scenarios?

Key findings

• All submitted systems outperformed the baseline in both speed and accuracy, demonstrating significant progress in efficient NMT.
• The Marian team’s systems consistently achieved the best trade-off between speed and accuracy, forming the Pareto frontier on both CPU and GPU.
• On CPU, self-attentional models (e.g., Marian-Trans-Small-AAN) were more efficient than RNN-based models, while on GPU, RNNs were faster but less accurate.
• The OpenNMT-Tiny system achieved only 220MB of CPU memory usage, making it the most memory-efficient, though with lower BLEU scores.
• Model distillation and low-precision computation (e.g., int8) were key enablers of efficiency, with quantized models showing significant speedups.
• The shared task established strong baselines for future research, highlighting the importance of engineering optimizations over novel architectures for efficiency.