[Paper Review] Testing Untestable Neural Machine Translation: An Industrial Case

Wujie Zheng, Wenyu Wang+7 more|arXiv (Cornell University)|Jul 6, 2018

Adversarial Robustness in Machine Learning21 references4 citations

TL;DR

This paper proposes a novel test oracle for in-vivo neural machine translation (NMT) testing that detects translation failures without requiring reference translations. By analyzing systematic properties of natural language—such as word/phrase consistency and structural coherence—across inputs and outputs, the approach identifies violations indicative of translation errors, achieving high effectiveness in real-world deployment at WeChat with over one billion users.

ABSTRACT

Neural Machine Translation (NMT) has been widely adopted recently due to its advantages compared with the traditional Statistical Machine Translation (SMT). However, an NMT system still often produces translation failures due to the complexity of natural language and sophistication in designing neural networks. While in-house black-box system testing based on reference translations (i.e., examples of valid translations) has been a common practice for NMT quality assurance, an increasingly critical industrial practice, named in-vivo testing, exposes unseen types or instances of translation failures when real users are using a deployed industrial NMT system. To fill the gap of lacking test oracle for in-vivo testing of an NMT system, in this paper, we propose a new approach for automatically identifying translation failures, without requiring reference translations for a translation task; our approach can directly serve as a test oracle for in-vivo testing. Our approach focuses on properties of natural language translation that can be checked systematically and uses information from both the test inputs (i.e., the texts to be translated) and the test outputs (i.e., the translations under inspection) of the NMT system. Our evaluation conducted on real-world datasets shows that our approach can effectively detect targeted property violations as translation failures. Our experiences on deploying our approach in both production and development environments of WeChat (a messenger app with over one billion monthly active users) demonstrate high effectiveness of our approach along with high industry impact.

Motivation & Objective

• To address the lack of test oracles for in-vivo testing of industrial NMT systems, where reference translations are unavailable.
• To detect translation failures in deployed NMT systems without relying on reference translations or quality scores like BLEU.
• To identify systematic violations of natural language translation properties that signal potential defects in model design, implementation, or training data.
• To enable real-time failure detection and diagnosis in production environments, improving translation quality and system reliability.
• To support both in-house model improvement and cross-system diagnosis of competing NMT providers.

Proposed method

• The approach constructs a word/phrase translation dictionary using item-based collaborative filtering on real user translation requests, leveraging co-occurrence patterns across source and target texts.
• It identifies translation failures by detecting violations of systematic linguistic properties, such as inconsistent word/phrase mappings or structural anomalies in translations.
• The method performs statistical analysis on both input texts (source sentences) and output translations, comparing their linguistic patterns to detect anomalies.
• It uses a dictionary of 130,000 English and 180,000 Chinese meaningful words/phrases derived from training data to validate translation consistency.
• The approach applies heuristics based on linguistic regularity and context coherence to flag translations that deviate from expected patterns, even without reference outputs.
• It is deployed in both production and development environments to monitor real-time user requests and detect issues at scale.

Experimental results

Research questions

• RQ1 Can translation failures be detected in industrial NMT systems without access to reference translations?
• RQ2 What systematic linguistic properties can serve as reliable indicators of translation defects in the absence of reference outputs?
• RQ3 How effective is the proposed method in identifying real-world translation failures in production-scale NMT systems?
• RQ4 Can the approach detect model-level issues such as design flaws or data quality problems in NMT systems?
• RQ5 To what extent can the method be generalized across different NMT providers and translation systems?

Key findings

• The approach successfully detected targeted property violations in translations from multiple industrial NMT providers, including issues like character-by-character translation and incorrect handling of hyphenated or abbreviated terms.
• It identified specific defects such as over-translation of common food terms (e.g., 'Runeberg torte' for 'cake') and incorrect abbreviation handling (e.g., 'p.' for 'plasmodium').
• The method enabled the collection of 130,000 English and 180,000 Chinese meaningful words/phrases from real user data, which were used as in-house test cases for model improvement.
• Deployment at WeChat demonstrated high effectiveness, with the approach detecting issues in both WeChat’s own NMT system and competing systems from other providers.
• The approach revealed model design flaws (e.g., disregard for context in multi-character phrases) and implementation issues (e.g., failure to process hyphenated terms), indicating diagnostic value beyond simple quality scoring.
• The method proved effective in diagnosing issues stemming from training data quality, such as overfitting to rare or specific term variants.