[Paper Review] THUMT: An Open Source Toolkit for Neural Machine Translation

Jiacheng Zhang, Yanzhuo Ding+5 more|arXiv (Cornell University)|Jun 20, 2017

Natural Language Processing TechniquesComputer Science15 references88 citations

TL;DR

THUMT presents an open-source NMT toolkit built on Theano, supporting MLE, MRT, and SST training criteria, with a visualization tool and unknown word replacement, showing competitive Chinese-English translation results.

ABSTRACT

This paper introduces THUMT, an open-source toolkit for neural machine translation (NMT) developed by the Natural Language Processing Group at Tsinghua University. THUMT implements the standard attention-based encoder-decoder framework on top of Theano and supports three training criteria: maximum likelihood estimation, minimum risk training, and semi-supervised training. It features a visualization tool for displaying the relevance between hidden states in neural networks and contextual words, which helps to analyze the internal workings of NMT. Experiments on Chinese-English datasets show that THUMT using minimum risk training significantly outperforms GroundHog, a state-of-the-art toolkit for NMT.

Motivation & Objective

• Motivate open-source development of NMT tools with flexible training criteria.
• Provide an encoder-decoder attention-based NMT implementation on Theano.
• Enable analysis of NMT internals via a visualization tool using layer-wise relevance propagation.
• Show performance and training-time trade-offs across training criteria and optimizers on Chinese-English translation.
• Demonstrate the benefits of semi-supervised and minimum-risk training for translation quality.

Proposed method

• Implement standard attention-based encoder-decoder NMT on Theano.
• Support three training criteria: maximum likelihood estimation (MLE), minimum risk training (MRT), and semi-supervised training (SST).
• Provide optimization options: SGD, Adadelta, and Adam (modified to avoid NaN).
• Offer a visualization tool based on layer-wise relevance propagation to analyze translations.
• Use FastAlign to build bilingual dictionaries for unknown word replacement.
• Compare THUMT with GroundHog on Chinese–English translation and report BLEU and training-time metrics.

Experimental results

Research questions

• RQ1 Does THUMT achieve competitive BLEU scores against a leading open-source NMT toolkit on Chinese–English translation?
• RQ2 What is the impact of MRT and SST on translation quality compared to standard MLE?
• RQ3 How do different optimizers affect translation performance and training efficiency in THUMT?
• RQ4 Can visualization of NMT internals help in understanding translation processes and diagnosing errors?
• RQ5 What is the effect of unknown word replacement on translation quality across criteria?

Key findings

• THUMT with MRT significantly improves over MLE on Chinese–English translation.
• Adam optimizer yields consistent improvements over AdaDelta for THUMT.
• SST leveraging monolingual corpora improves translation quality in both directions (zh→en, en→zh).
• Replacing unknown words consistently improves results across criteria and optimizers.
• Training time varies substantially by criterion and optimizer, with MLE+Adam being faster than MRT; SST is relatively efficient among non-MLE criteria.