[Paper Review] OpenNMT: Neural Machine Translation Toolkit

Guillaume Klein, Yoon Kim+4 more|arXiv (Cornell University)|May 28, 2018

Natural Language Processing Techniques30 references51 citations

TL;DR

OpenNMT presents an open-source, modular, and efficient toolkit for neural machine translation with multiple implementations (Lua/Torch, PyTorch, TensorFlow) aimed at research and production use. It emphasizes extensibility, speed, and broad support for modalities and attention mechanisms.

ABSTRACT

OpenNMT is an open-source toolkit for neural machine translation (NMT). The system prioritizes efficiency, modularity, and extensibility with the goal of supporting NMT research into model architectures, feature representations, and source modalities, while maintaining competitive performance and reasonable training requirements. The toolkit consists of modeling and translation support, as well as detailed pedagogical documentation about the underlying techniques. OpenNMT has been used in several production MT systems, modified for numerous research papers, and is implemented across several deep learning frameworks.

Motivation & Objective

• Introduce OpenNMT as an open-source NMT toolkit prioritizing efficiency, modularity, and extensibility for research and production.
• Describe implementation details, design goals, and performance benchmarks across frameworks.
• Demonstrate the toolkit's applicability to various input modalities and modern NMT techniques.
• Highlight its ecosystem, documentation, and community adoption.

Proposed method

• Present three main implementations: OpenNMT-lua (Torch 7), OpenNMT-py (PyTorch), and OpenNMT-tf (TensorFlow).
• Describe memory sharing, sharding, and multi-GPU data-parallel training to improve efficiency.
• Explain modular architecture with clear interfaces (preprocess, train, translate) and configurable components.
• Detail support for attention variants, copying mechanisms, beam search, and OOV handling.
• Explain extensions to non-sequential inputs (tables, speech, images) via encoders and support for Transformer and convolutional models.

Experimental results

Research questions

• RQ1 How can an open-source NMT toolkit balance training efficiency with modularity and extensibility?
• RQ2 What are the design choices and implementations that enable production-ready performance across multiple frameworks?
• RQ3 How effectively does OpenNMT support various input modalities and recent NMT innovations (attention variants, copy mechanisms, Transformer, etc.)?
• RQ4 What benchmarks show OpenNMT's competitiveness in accuracy, speed, and memory usage against contemporary systems?

Key findings

• OpenNMT achieves competitive translation quality on standard benchmarks relative to contemporaries.
• The toolkit demonstrates substantial memory and speed optimizations, enabling larger batch sizes and multi-GPU training.
• OpenNMT supports a wide range of input modalities and model variants, including attention types and transformer models.
• It provides extensible, readable code with concise core implementations across Lua, PyTorch, and TensorFlow.
• Numerous production deployments and research adaptations have utilized OpenNMT since its release.