[Paper Review] Understanding Pre-Editing for Black-Box Neural Machine Translation

Rei Miyata, Atsushi Fujita|arXiv (Cornell University)|Feb 5, 2021

Natural Language Processing Techniques28 references4 citations

TL;DR

This study investigates human pre-editing practices for black-box neural machine translation (NMT) by collecting 6,652 pre-editing instances across three translation directions, two NMT systems, and four text domains. It reveals that enhancing semantic clarity and syntactic explicitness is more effective than simplifying or shortening source texts, and identifies predictable patterns in how different editing operations affect NMT output quality despite overall unpredictability.

ABSTRACT

Pre-editing is the process of modifying the source text (ST) so that it can be translated by machine translation (MT) in a better quality. Despite the unpredictability of black-box neural MT (NMT), pre-editing has been deployed in various practical MT use cases. Although many studies have demonstrated the effectiveness of pre-editing methods for particular settings, thus far, a deep understanding of what pre-editing is and how it works for black-box NMT is lacking. To elicit such understanding, we extensively investigated human pre-editing practices. We first implemented a protocol to incrementally record the minimum edits for each ST and collected 6,652 instances of pre-editing across three translation directions, two MT systems, and four text domains. We then analysed the instances from three perspectives: the characteristics of the pre-edited ST, the diversity of pre-editing operations, and the impact of the pre-editing operations on NMT outputs. Our findings include the following: (1) enhancing the explicitness of the meaning of an ST and its syntactic structure is more important for obtaining better translations than making the ST shorter and simpler, and (2) although the impact of pre-editing on NMT is generally unpredictable, there are some tendencies of changes in the NMT outputs depending on the editing operation types.

Motivation & Objective

• To understand how human pre-editing improves translation quality in black-box NMT systems.
• To investigate the characteristics of pre-edited source texts and their impact on NMT outputs.
• To analyze the diversity and effects of different pre-editing operations on NMT performance.
• To provide empirical insights for developing robust pre-editing strategies and automatic tools for NMT.

Proposed method

• A human-in-the-loop protocol was implemented to incrementally record the minimum necessary edits for 6,652 source text instances across three translation directions.
• Pre-editing was collected across two NMT systems (Google and Marian NMT) and four text domains (e.g., news, technical, medical, legal).
• The study analyzed pre-edited source texts for linguistic characteristics such as length, syntactic complexity, and explicitness of meaning.
• Editing operations were categorized into structural, functional, phrase-level, and orthographic types, and their impacts on NMT outputs were measured using TER (Translation Edit Rate).
• Statistical correlation analysis was performed between changes in source text and changes in MT output to assess predictability.
• The impact of each editing operation type was evaluated based on the magnitude of change in the NMT output, using TER as the primary metric.

Experimental results

Research questions

• RQ1 What linguistic characteristics of pre-edited source texts most strongly correlate with improved NMT output quality?
• RQ2 How do different types of editing operations (e.g., structural, phrase reordering, punctuation changes) affect NMT output changes?
• RQ3 To what extent is the impact of pre-editing on NMT outputs predictable across different translation directions and MT systems?
• RQ4 Are there consistent patterns in how specific editing operations influence translation quality, despite the inherent unpredictability of NMT?

Key findings

• Enhancing the explicitness of meaning and syntactic structure in the source text is more effective for improving NMT quality than reducing length or simplifying sentences.
• The impact of pre-editing on NMT outputs is generally unpredictable, but certain editing operation types show consistent trends in their effects on translation output.
• Structural edits (e.g., reordering clauses, changing sentence structure) caused the largest changes in NMT output, as measured by TER.
• Phrase reordering (P03) had a relatively small impact on NMT output, indicating that modern NMT systems preserve phrase-level equivalence across position shifts.
• The use or removal of chunking markers (P02) significantly affected NMT output, likely due to tokenization changes, despite minimal impact on human readability.
• A moderate positive correlation (Pearson’s r = 0.580, Spearman’s ρ = 0.574) was observed between source text edits and NMT output changes in Japanese-to-Korean translation, suggesting higher predictability in closely related language pairs.