[Paper Review] Achieving Human Parity on Automatic Chinese to English News Translation

Hany Hassan, Anthony Aue+22 more|arXiv (Cornell University)|Mar 15, 2018

Natural Language Processing Techniques17 references578 citations

TL;DR

The paper defines human parity for translation, builds a Chinese→English neural MT system with dual learning, deliberation networks, and agreement regularization, and demonstrates parity with professional human translations on WMT17 news data.

ABSTRACT

Machine translation has made rapid advances in recent years. Millions of people are using it today in online translation systems and mobile applications in order to communicate across language barriers. The question naturally arises whether such systems can approach or achieve parity with human translations. In this paper, we first address the problem of how to define and accurately measure human parity in translation. We then describe Microsoft's machine translation system and measure the quality of its translations on the widely used WMT 2017 news translation task from Chinese to English. We find that our latest neural machine translation system has reached a new state-of-the-art, and that the translation quality is at human parity when compared to professional human translations. We also find that it significantly exceeds the quality of crowd-sourced non-professional translations.

Motivation & Objective

• Define and measure human parity in machine translation for the news domain.
• Develop a neural machine translation system for Chinese→English that reaches parity with human translations.
• Leverage dual learning, joint training, and data filtering to improve translation quality.

Proposed method

• Adopt source-based direct assessment for human evaluation to avoid reference bias.
• Use Transformer-based NMT as the base architecture for Zh→En translation.
• Implement dual learning to exploit both source→target and target→source models for supervised and unsupervised data.
• Apply joint training with back-translation and semi-supervised objectives to leverage monolingual data in both directions.
• Incorporate two approaches to mitigate exposure bias: Deliberation Networks (two-pass decoding) and agreement regularization between left-to-right and right-to-left models.
• Perform data selection and filtering to improve training data quality.
• Demonstrate system combination to leverage complementary strengths of multiple models.

Experimental results

Research questions

• RQ1 What is a principled definition of human parity in translation and how can it be measured statistically?
• RQ2 Can a Chinese→English MT system reach parity with professional human translations on news domain data?
• RQ3 How much do dual learning, joint training, and bidirectional data leveraging contribute to translation quality?
• RQ4 Do two-pass decoding and cross-direction agreement improve generation quality and reduce exposure bias?
• RQ5 How does data quality and system combination affect approaching human parity?

Key findings

• The latest neural MT system achieves state-of-the-art quality on Chinese→English news translation.
• The system’s quality is statistically indistinguishable from professional human translations on the evaluated test set.
• The approach significantly exceeds crowd-sourced non-professional translations.
• Dual learning and joint training effectively utilize monolingual and bilingual data.
• Deliberation networks and agreement regularization help mitigate exposure bias and improve translation quality.
• System combination yields further gains by leveraging complementary model strengths.